502 Bad Gateway (nginx) after Update to 2.3
-
It just happened again. "Restart php-fpm" solved it, but there is definitely some bug.
-
- Have same issue on 2.3.4
- Restarted PHP-FPM restored GUI and OpenVPN
- Removed IPsec widget from dashboard, hopefully will help
-
Hello,
I'm getting the 502 bad gateway too. I have just installed pfsense 2.3.4.
I start getting the issue after setting up four VLANs on my OPT1 interface. Using option 16 Restart PHP-FPM is working sometimes on first shot but more frequently after second or third attempt.
CPU usage was arround 1.5%Today I install SQUID and activate transparent proxy mode on my LAN and four OPT1 VLANs.
After I login in webconfigurator, I cannot access any page. I immediately fall on the nginx 502 bad gateway error. I have tried to use option 16 as before but it does not work anymore. Only a reboot of pfSense box allow me to login again in the web configurator.After several reboots I saw that sometimes the OPT1 appears down and the VLANs are up, , sometimes OPT1 and VLANs are down, sometimes everything is up as expected :-[
Each time CPU usage increase to 100% and then in a few seconds I got the 502 message.I have made some tests and when I disconnect cable on OPT1, issue does not happen.
I have checked option "Do not forward traffic to Private Address Space (RFC 1918) destinations" and plug in the cable on opt1. It lloks like I am back to initial situation now.It fails less often but is still annoying since I have to reboot the box to continue my configuration.
If you have any suggestion with this I will really apreciate. For now, I will stay on pfSense 2.2
-
Ι can confirm the issue with 2.3.4-RELEASE (amd64). When the gui is not accessible, OpenVPN users cannot login. When I used option 16, the users could login again.
I disabled the widgets mentioned before (although I need the OpenVPN widget…) and see what's happen.
Best regards
Kostas
-
I believe that the root of all my problems has been an auto-negotiate failure on the WAN interface …
Update: I'm convinced that my problems all stem from using a cheap Chinese "pfSense" system that I purchased on Amazon - it was about half the price of a comparable Netgate unit but it has continuously generated errors on the WAN interface and has never managed to auto-negotiate the link speed. Recently I had the cable company at the house after complaining that the cable speed (at 10Mbs) was too low - I'm paying for 150Mbs. After about an hour of trying everything and failing to fix the problem I put a switch in between the cable modem and the firewall wan port - which boosted the speed to about 55Mbs.
I have replaced the "el cheapo" Chinese box with a Netgate box - auto-negotiated works, the cable modem instantly supports 1Gbs and I'm not getting 160Mbs on the cable connection. I'm no longer seeing any errors on the WAN interface.
-
sunday i updated 6 firewalls from 2.3 to 2.4.
All suddenly are showing 502 bad gateway problems.
Besides that 2 of them become unresponsive after around 20 hours.
Its hard to access the firewall then but the one time i got lucky i can see that the memory is completely full.
So i doubled the memory and still same problem occur.Only way to solve it is rebooting the firewall
btw, all are vm's
2 of them were a complete new install with a config restore (i had to go from 32bits to 64 bits).My physical pfsense's are not infected…
-
Search in this forum or on redmine.pfsense.org - this was seen before.
IIRC it may come from an associated CD/DVD drive to your pfSense VM. Get rid of that and it might work. Other problem I don't remember off of my head is discussed on redmine. -
sunday i updated 6 firewalls from 2.3 to 2.4.
All suddenly are showing 502 bad gateway problems.
Besides that 2 of them become unresponsive after around 20 hours.
Its hard to access the firewall then but the one time i got lucky i can see that the memory is completely full.
So i doubled the memory and still same problem occur.Only way to solve it is rebooting the firewall
btw, all are vm's
2 of them were a complete new install with a config restore (i had to go from 32bits to 64 bits).My physical pfsense's are not infected…
Exactly same problem here! Firewall dies, and 502 bad gateway.
-
Hello,
I experienced the same problem on 2.4.0 release.
I have no access to my firewall by :- SSH
- WebGUI
- Console
BUT, I still continue to surf on the Internet… Very strange...
This issue occurs at the end of 20 hours uptime.
It's hardening to identify the root cause.
The only way to resolve this problem is rebooting my hardware appliance.
Once the reboot process is done, there is no event logs...I have the following widget on my dashboard :
- System Informations
- NTP Status
- SMART Status
- pfBlockerNG
- OpenVPN
- Gateways
- Interfaces
- Interfaces Statistics
- Traffic Graphs
- Services status
- Firewall logs
- Thermal sensors
Heres my hardware configuration :
- MotherBoard : APU2C4
- CPU : AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support
- RAM : 4 GByte DDR3-1333 DRAM
- Storage Type : mSATA SSD
- Storage Size : 120GB
- Ethernet ports : 3 x 1Gbit/s
- Wireless card : WLE200NX with two antenna
-
Update pfBlockerNG to v2.1.2 and reboot.
https://forum.pfsense.org/index.php?topic=137103.0
-
Hi BBcan177,
Many thanks.
I was already in this version. But, I deleted some widget and increased the number of webconfigurator process up to 4.
I also disabled autocheck updates in Dashboard and deinstalled NTOPNG due to some unstabilities.
I keep an eye on it. -
This has reared its ugly head again on the latest 2.4.1 release. I have about 10% of my FW's doing this.
I tried to update the packages:
pkg update Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date.On the webUI it sits and spins.
So…apparently the repo's are really down and you can't get to them?
https://forum.pfsense.org/index.php?topic=139276.0
I updated two of the bad apples today in hopes they'd be fixed form the nginx problem.
I can't ssh to it, when I login it just says "pfsense -" <-- I can't do the /etc/init....whatever to start the console number list, I can't shutdown -r now, nothing...it just sites there. I also can't OpenVPN into the box which is when I start getting the phone calls then someone has to go in and reboot the firewall.
Is this PFBng doing this? Perish the thought! :0)
1 - When will the repos be back up?
2 - I swear I remembered to update the packages on each box after updating but maybe I missed these twoOnce again...thanks PFSense for all you do, the ONLY firewall I use and deploy in production!
-
Try to hit: CTRL - z and then enter "/bin/tcsh"
-
I"ve been seeing this off and on with 2.4. It seemed to be related to a pfBlockerNG bug that was patched – but with 2.4.2 it's happening again. I tried the Control-Z trick as suggested and get no response
-
I did briefly look into this and have a theory of whats going on, but its only a theory.
pfSense backend scripts, a lot of them run on PHP.
Instead of using PHP cli, they seem to utilise PHP FPM instead of PHP CLI for the cli scripts, which in turn will utilise the FPM server processes, if you have some of these background scripts been processed "whilst" trying to access the GUI, then none may be available for nginx to use and you will get the gateway error.
If I am correct there is a number of possible resolutions, each with their own downsides.
1 - Stop using FPM to run background PHP stuff.
2 - Increase FPM processes.
3 - Retune FPM so its more scalable for higher load and avoid gateway errors. -
Posting a patch which should improve the issue. (after applying the patch choose option 16 to restart php-fpm)
The downsides of the changes are that memory consumption will be higher, but on my unit most ram is unused anyway. Stock pfSense seems to be tuned for the very low end stuff.
--- /etc/rc.php_ini_setup 2017-07-06 19:35:29.000000000 +0100 +++ /etc/rc.php_ini_setup 2017-11-22 21:05:24.986893000 +0000 @@ -265,10 +265,22 @@ PHPFPMMAX=3 +PHPFPMIDLE=30 +PHPFPMSTART=1 +PHPFPMSPARE=2 +PHPFPMREQ=500 if [ $REALMEM -lt 250 ]; then PHPFPMMAX=2 + PHPFPMIDLE=5 + PHPFPMSTART=1 + PHPFPMSPARE=1 + PHPFPMREQ=500 elif [ ${REALMEM} -gt 1000 ]; then - PHPFPMMAX=4 + PHPFPMMAX=8 + PHPFPMIDLE=3600 + PHPFPMSTART=2 + PHPFPMSPARE=7 + PHPFPMREQ=5000 fi /bin/cat > /usr/local/lib/php-fpm.conf <<eof<br>@@ -305,9 +317,9 @@ /bin/cat >> /usr/local/lib/php-fpm.conf < <eof<br>pm = ondemand -pm.process_idle_timeout = 5 +pm.process_idle_timeout = $PHPFPMIDLE pm.max_children = $PHPFPMMAX -pm.max_requests = 500 +pm.max_requests = $PHPFPMREQ EOF @@ -315,12 +327,12 @@ /bin/cat >> /usr/local/lib/php-fpm.conf < <eof<br>pm = dynamic -pm.process_idle_timeout = 5 +pm.process_idle_timeout = $PHPFPMIDLE pm.max_children = $PHPFPMMAX -pm.start_servers = 1 -pm.max_requests = 500 +pm.start_servers = $PHPFPMSTART +pm.max_requests = $PHPFPMREQ pm.min_spare_servers=1 -pm.max_spare_servers=1 +pm.max_spare_servers= $PHPFPMSPARE EOF else @@ -329,7 +341,7 @@ pm = static pm.max_children = $PHPFPMMAX -pm.max_requests = 500 +pm.max_requests = $PHPFPMREQ EOF</eof<br></eof<br></eof<br> -
Chris had asked me to to create a git commit for this and I had not done so, I now have. So if any would like to try his fix here is the patch ID: 2c131b1
-
Thanks martin.
Quick update here also.
https://forum.pfsense.org/index.php?topic=137103.msg767939#msg767939
Positive feedback from myself.
-
The patch in message #60 worked for me. After upgrading to 2.4.2 I tried installing PFBlockerNG and started getting non-responsive web interface, ssh sessions and even serial console. I am pretty new to BSD and couldn't get the patch to take, but I got the changes in manually and my router is stable again. Created an account to say, "Thank you!"
-
Odd, I created the patch for Chris's mods and PR'd it for him. Are you saying that Patch ID: 028be76 will not apply for you?
I have just removed it and re-fetched it and re-applied it to my system without any problems.
