502 Bad Gateway (nginx) after Update to 2.3
-
sunday i updated 6 firewalls from 2.3 to 2.4.
All suddenly are showing 502 bad gateway problems.
Besides that 2 of them become unresponsive after around 20 hours.
Its hard to access the firewall then but the one time i got lucky i can see that the memory is completely full.
So i doubled the memory and still same problem occur.Only way to solve it is rebooting the firewall
btw, all are vm's
2 of them were a complete new install with a config restore (i had to go from 32bits to 64 bits).My physical pfsense's are not infected…
-
Search in this forum or on redmine.pfsense.org - this was seen before.
IIRC it may come from an associated CD/DVD drive to your pfSense VM. Get rid of that and it might work. Other problem I don't remember off of my head is discussed on redmine. -
sunday i updated 6 firewalls from 2.3 to 2.4.
All suddenly are showing 502 bad gateway problems.
Besides that 2 of them become unresponsive after around 20 hours.
Its hard to access the firewall then but the one time i got lucky i can see that the memory is completely full.
So i doubled the memory and still same problem occur.Only way to solve it is rebooting the firewall
btw, all are vm's
2 of them were a complete new install with a config restore (i had to go from 32bits to 64 bits).My physical pfsense's are not infected…
Exactly same problem here! Firewall dies, and 502 bad gateway.
-
Hello,
I experienced the same problem on 2.4.0 release.
I have no access to my firewall by :- SSH
- WebGUI
- Console
BUT, I still continue to surf on the Internet… Very strange...
This issue occurs at the end of 20 hours uptime.
It's hardening to identify the root cause.
The only way to resolve this problem is rebooting my hardware appliance.
Once the reboot process is done, there is no event logs...I have the following widget on my dashboard :
- System Informations
- NTP Status
- SMART Status
- pfBlockerNG
- OpenVPN
- Gateways
- Interfaces
- Interfaces Statistics
- Traffic Graphs
- Services status
- Firewall logs
- Thermal sensors
Heres my hardware configuration :
- MotherBoard : APU2C4
- CPU : AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support
- RAM : 4 GByte DDR3-1333 DRAM
- Storage Type : mSATA SSD
- Storage Size : 120GB
- Ethernet ports : 3 x 1Gbit/s
- Wireless card : WLE200NX with two antenna
-
Update pfBlockerNG to v2.1.2 and reboot.
https://forum.pfsense.org/index.php?topic=137103.0
-
Hi BBcan177,
Many thanks.
I was already in this version. But, I deleted some widget and increased the number of webconfigurator process up to 4.
I also disabled autocheck updates in Dashboard and deinstalled NTOPNG due to some unstabilities.
I keep an eye on it. -
This has reared its ugly head again on the latest 2.4.1 release. I have about 10% of my FW's doing this.
I tried to update the packages:
pkg update Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date.On the webUI it sits and spins.
So…apparently the repo's are really down and you can't get to them?
https://forum.pfsense.org/index.php?topic=139276.0
I updated two of the bad apples today in hopes they'd be fixed form the nginx problem.
I can't ssh to it, when I login it just says "pfsense -" <-- I can't do the /etc/init....whatever to start the console number list, I can't shutdown -r now, nothing...it just sites there. I also can't OpenVPN into the box which is when I start getting the phone calls then someone has to go in and reboot the firewall.
Is this PFBng doing this? Perish the thought! :0)
1 - When will the repos be back up?
2 - I swear I remembered to update the packages on each box after updating but maybe I missed these twoOnce again...thanks PFSense for all you do, the ONLY firewall I use and deploy in production!
-
Try to hit: CTRL - z and then enter "/bin/tcsh"
-
I"ve been seeing this off and on with 2.4. It seemed to be related to a pfBlockerNG bug that was patched – but with 2.4.2 it's happening again. I tried the Control-Z trick as suggested and get no response
-
I did briefly look into this and have a theory of whats going on, but its only a theory.
pfSense backend scripts, a lot of them run on PHP.
Instead of using PHP cli, they seem to utilise PHP FPM instead of PHP CLI for the cli scripts, which in turn will utilise the FPM server processes, if you have some of these background scripts been processed "whilst" trying to access the GUI, then none may be available for nginx to use and you will get the gateway error.
If I am correct there is a number of possible resolutions, each with their own downsides.
1 - Stop using FPM to run background PHP stuff.
2 - Increase FPM processes.
3 - Retune FPM so its more scalable for higher load and avoid gateway errors. -
Posting a patch which should improve the issue. (after applying the patch choose option 16 to restart php-fpm)
The downsides of the changes are that memory consumption will be higher, but on my unit most ram is unused anyway. Stock pfSense seems to be tuned for the very low end stuff.
--- /etc/rc.php_ini_setup 2017-07-06 19:35:29.000000000 +0100 +++ /etc/rc.php_ini_setup 2017-11-22 21:05:24.986893000 +0000 @@ -265,10 +265,22 @@ PHPFPMMAX=3 +PHPFPMIDLE=30 +PHPFPMSTART=1 +PHPFPMSPARE=2 +PHPFPMREQ=500 if [ $REALMEM -lt 250 ]; then PHPFPMMAX=2 + PHPFPMIDLE=5 + PHPFPMSTART=1 + PHPFPMSPARE=1 + PHPFPMREQ=500 elif [ ${REALMEM} -gt 1000 ]; then - PHPFPMMAX=4 + PHPFPMMAX=8 + PHPFPMIDLE=3600 + PHPFPMSTART=2 + PHPFPMSPARE=7 + PHPFPMREQ=5000 fi /bin/cat > /usr/local/lib/php-fpm.conf <<eof<br>@@ -305,9 +317,9 @@ /bin/cat >> /usr/local/lib/php-fpm.conf < <eof<br>pm = ondemand -pm.process_idle_timeout = 5 +pm.process_idle_timeout = $PHPFPMIDLE pm.max_children = $PHPFPMMAX -pm.max_requests = 500 +pm.max_requests = $PHPFPMREQ EOF @@ -315,12 +327,12 @@ /bin/cat >> /usr/local/lib/php-fpm.conf < <eof<br>pm = dynamic -pm.process_idle_timeout = 5 +pm.process_idle_timeout = $PHPFPMIDLE pm.max_children = $PHPFPMMAX -pm.start_servers = 1 -pm.max_requests = 500 +pm.start_servers = $PHPFPMSTART +pm.max_requests = $PHPFPMREQ pm.min_spare_servers=1 -pm.max_spare_servers=1 +pm.max_spare_servers= $PHPFPMSPARE EOF else @@ -329,7 +341,7 @@ pm = static pm.max_children = $PHPFPMMAX -pm.max_requests = 500 +pm.max_requests = $PHPFPMREQ EOF</eof<br></eof<br></eof<br> -
Chris had asked me to to create a git commit for this and I had not done so, I now have. So if any would like to try his fix here is the patch ID: 2c131b1
-
Thanks martin.
Quick update here also.
https://forum.pfsense.org/index.php?topic=137103.msg767939#msg767939
Positive feedback from myself.
-
The patch in message #60 worked for me. After upgrading to 2.4.2 I tried installing PFBlockerNG and started getting non-responsive web interface, ssh sessions and even serial console. I am pretty new to BSD and couldn't get the patch to take, but I got the changes in manually and my router is stable again. Created an account to say, "Thank you!"
-
Odd, I created the patch for Chris's mods and PR'd it for him. Are you saying that Patch ID: 028be76 will not apply for you?
I have just removed it and re-fetched it and re-applied it to my system without any problems.
-
It's probably something i did wrong. Like I said I'm not familiar with BSD, and only used patch a couple of times.
Saved the patch listed in this thread as rc.php_ini_setup.patch
Here's a recreation of what I did.
/etc: patch rc.php_ini_setup rc.php_ini_setup.patch
Hmm… Looks like a unified diff to me...
The text leading up to this was:-- /etc/rc.php_ini_setup 2017-07-06 19:35:29.000000000 +0100
+++ /etc/rc.php_ini_setup 2017-11-22 21:05:24.986893000 +0000 Patching file rc.php_ini_setup using Plan A... Hunk #1 failed at 265. Hunk #2 succeeded at 317 with fuzz 1. Hunk #3 succeeded at 327 with fuzz 1. Hunk #4 succeeded at 341. 1 out of 4 hunks failed--saving rejects to rc.php_ini_setup.rej done -rw-r--r-- 1 root wheel 1366 Nov 28 05:58 rc.php_ini_setup.patch
-rwxr-xr-x 1 root wheel 9395 Nov 28 05:58 rc.php_ini_setup.orig
-rwxr-xr-x 1 root wheel 9459 Nov 28 05:59 rc.php_ini_setup
-rw------- 1 root wheel 456 Nov 28 05:59 rc.php_ini_setup.rej/etc: cat rc.php_ini_setup.rej
@@ -265,10 +265,22 @@PHPFPMMAX=3
+PHPFPMIDLE=30
+PHPFPMSTART=1
+PHPFPMSPARE=2
+PHPFPMREQ=500
if [ $REALMEM -lt 250 ]; then
PHPFPMMAX=2
+ PHPFPMIDLE=5
+ PHPFPMSTART=1
+ PHPFPMSPARE=1
+ PHPFPMREQ=500
elif [ ${REALMEM} -gt 1000 ]; then
- PHPFPMMAX=4
+ PHPFPMMAX=8
+ PHPFPMIDLE=3600
+ PHPFPMSTART=2
+ PHPFPMSPARE=7
+ PHPFPMREQ=5000
fi/bin/cat > /usr/local/lib/php-fpm.conf < <eof<br>/etc: diff rc.php_ini_setup.orig rc.php_ini_setup
308c308
< pm.process_idle_timeout = 5
–-pm.process_idle_timeout = $PHPFPMIDLE
310c310
< pm.max_requests = 500
pm.max_requests = $PHPFPMREQ
318c318
< pm.process_idle_timeout = 5
pm.process_idle_timeout = $PHPFPMIDLE
320,321c320,321
< pm.start_servers = 1
< pm.max_requests = 500
pm.start_servers = $PHPFPMSTART
pm.max_requests = $PHPFPMREQ
323c323
< pm.max_spare_servers=1
pm.max_spare_servers= $PHPFPMSPARE
332c332
< pm.max_requests = 500
pm.max_requests = $PHPFPMREQ</eof<br>
After that I just restored the saved copy and just manually make the changes. Still working great. I'm using a PC engines apu2c2 and memory usage is about 12 percent Thanks again.
-
It's probably something i did wrong. Like I said I'm not familiar with BSD, and only used patch a couple of times.
Saved the patch listed in this thread as rc.php_ini_setup.patch
Use the System_Patches package, makes it a lot easier.
-
Ah, not familiar with the pfSense patch system. :)
OK, first thing to do is reverse what you have done, put everything back as it was.
Next, go into pfSense and select Package Manager from the System Menu, select 'Available Packages' and find the 'System Patches' package and install it.
You will now have a 'Patches' option in the system menu - Select it.
Click on add new patch.
In the URL/Commit ID enter the patch ID 028be76, give a description in the Description box and press save.
Now click on Test, it should say can be applied cleanly. Now click on Apply.
Job done.
-
Thank you again. I replaced the original file and re-applied the patch the proper way for pfSense. Other than some extra spaces I had it correct.
The router continues to be stable with this patch. -
That's good.
The advantage of using the patch system, apart from saving you a lot of work, is that you have a setting in the patch that allows for it to be automatically re-applied when you update pfSense. It checks to see whether it can be re-applied cleanly and does so. Obviously, if the file(s) the patches are applied against change in a way that means the patch will not cleanly apply then it does not do so, so when you use patches it's advisable to check after an update. For the average user who is not on the beta trail this means only a two or three times a year, beta testers should check every time they update.
