502 Bad Gateway (nginx) after Update to 2.3
-
Hello,
I experienced the same problem on 2.4.0 release.
I have no access to my firewall by :- SSH
- WebGUI
- Console
BUT, I still continue to surf on the Internet… Very strange...
This issue occurs at the end of 20 hours uptime.
It's hardening to identify the root cause.
The only way to resolve this problem is rebooting my hardware appliance.
Once the reboot process is done, there is no event logs...I have the following widget on my dashboard :
- System Informations
- NTP Status
- SMART Status
- pfBlockerNG
- OpenVPN
- Gateways
- Interfaces
- Interfaces Statistics
- Traffic Graphs
- Services status
- Firewall logs
- Thermal sensors
Heres my hardware configuration :
- MotherBoard : APU2C4
- CPU : AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support
- RAM : 4 GByte DDR3-1333 DRAM
- Storage Type : mSATA SSD
- Storage Size : 120GB
- Ethernet ports : 3 x 1Gbit/s
- Wireless card : WLE200NX with two antenna
-
Update pfBlockerNG to v2.1.2 and reboot.
https://forum.pfsense.org/index.php?topic=137103.0
-
Hi BBcan177,
Many thanks.
I was already in this version. But, I deleted some widget and increased the number of webconfigurator process up to 4.
I also disabled autocheck updates in Dashboard and deinstalled NTOPNG due to some unstabilities.
I keep an eye on it. -
This has reared its ugly head again on the latest 2.4.1 release. I have about 10% of my FW's doing this.
I tried to update the packages:
pkg update Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date.On the webUI it sits and spins.
So…apparently the repo's are really down and you can't get to them?
https://forum.pfsense.org/index.php?topic=139276.0
I updated two of the bad apples today in hopes they'd be fixed form the nginx problem.
I can't ssh to it, when I login it just says "pfsense -" <-- I can't do the /etc/init....whatever to start the console number list, I can't shutdown -r now, nothing...it just sites there. I also can't OpenVPN into the box which is when I start getting the phone calls then someone has to go in and reboot the firewall.
Is this PFBng doing this? Perish the thought! :0)
1 - When will the repos be back up?
2 - I swear I remembered to update the packages on each box after updating but maybe I missed these twoOnce again...thanks PFSense for all you do, the ONLY firewall I use and deploy in production!
-
Try to hit: CTRL - z and then enter "/bin/tcsh"
-
I"ve been seeing this off and on with 2.4. It seemed to be related to a pfBlockerNG bug that was patched – but with 2.4.2 it's happening again. I tried the Control-Z trick as suggested and get no response
-
I did briefly look into this and have a theory of whats going on, but its only a theory.
pfSense backend scripts, a lot of them run on PHP.
Instead of using PHP cli, they seem to utilise PHP FPM instead of PHP CLI for the cli scripts, which in turn will utilise the FPM server processes, if you have some of these background scripts been processed "whilst" trying to access the GUI, then none may be available for nginx to use and you will get the gateway error.
If I am correct there is a number of possible resolutions, each with their own downsides.
1 - Stop using FPM to run background PHP stuff.
2 - Increase FPM processes.
3 - Retune FPM so its more scalable for higher load and avoid gateway errors. -
Posting a patch which should improve the issue. (after applying the patch choose option 16 to restart php-fpm)
The downsides of the changes are that memory consumption will be higher, but on my unit most ram is unused anyway. Stock pfSense seems to be tuned for the very low end stuff.
--- /etc/rc.php_ini_setup 2017-07-06 19:35:29.000000000 +0100 +++ /etc/rc.php_ini_setup 2017-11-22 21:05:24.986893000 +0000 @@ -265,10 +265,22 @@ PHPFPMMAX=3 +PHPFPMIDLE=30 +PHPFPMSTART=1 +PHPFPMSPARE=2 +PHPFPMREQ=500 if [ $REALMEM -lt 250 ]; then PHPFPMMAX=2 + PHPFPMIDLE=5 + PHPFPMSTART=1 + PHPFPMSPARE=1 + PHPFPMREQ=500 elif [ ${REALMEM} -gt 1000 ]; then - PHPFPMMAX=4 + PHPFPMMAX=8 + PHPFPMIDLE=3600 + PHPFPMSTART=2 + PHPFPMSPARE=7 + PHPFPMREQ=5000 fi /bin/cat > /usr/local/lib/php-fpm.conf <<eof<br>@@ -305,9 +317,9 @@ /bin/cat >> /usr/local/lib/php-fpm.conf < <eof<br>pm = ondemand -pm.process_idle_timeout = 5 +pm.process_idle_timeout = $PHPFPMIDLE pm.max_children = $PHPFPMMAX -pm.max_requests = 500 +pm.max_requests = $PHPFPMREQ EOF @@ -315,12 +327,12 @@ /bin/cat >> /usr/local/lib/php-fpm.conf < <eof<br>pm = dynamic -pm.process_idle_timeout = 5 +pm.process_idle_timeout = $PHPFPMIDLE pm.max_children = $PHPFPMMAX -pm.start_servers = 1 -pm.max_requests = 500 +pm.start_servers = $PHPFPMSTART +pm.max_requests = $PHPFPMREQ pm.min_spare_servers=1 -pm.max_spare_servers=1 +pm.max_spare_servers= $PHPFPMSPARE EOF else @@ -329,7 +341,7 @@ pm = static pm.max_children = $PHPFPMMAX -pm.max_requests = 500 +pm.max_requests = $PHPFPMREQ EOF</eof<br></eof<br></eof<br> -
Chris had asked me to to create a git commit for this and I had not done so, I now have. So if any would like to try his fix here is the patch ID: 2c131b1
-
Thanks martin.
Quick update here also.
https://forum.pfsense.org/index.php?topic=137103.msg767939#msg767939
Positive feedback from myself.
-
The patch in message #60 worked for me. After upgrading to 2.4.2 I tried installing PFBlockerNG and started getting non-responsive web interface, ssh sessions and even serial console. I am pretty new to BSD and couldn't get the patch to take, but I got the changes in manually and my router is stable again. Created an account to say, "Thank you!"
-
Odd, I created the patch for Chris's mods and PR'd it for him. Are you saying that Patch ID: 028be76 will not apply for you?
I have just removed it and re-fetched it and re-applied it to my system without any problems.
-
It's probably something i did wrong. Like I said I'm not familiar with BSD, and only used patch a couple of times.
Saved the patch listed in this thread as rc.php_ini_setup.patch
Here's a recreation of what I did.
/etc: patch rc.php_ini_setup rc.php_ini_setup.patch
Hmm… Looks like a unified diff to me...
The text leading up to this was:-- /etc/rc.php_ini_setup 2017-07-06 19:35:29.000000000 +0100
+++ /etc/rc.php_ini_setup 2017-11-22 21:05:24.986893000 +0000 Patching file rc.php_ini_setup using Plan A... Hunk #1 failed at 265. Hunk #2 succeeded at 317 with fuzz 1. Hunk #3 succeeded at 327 with fuzz 1. Hunk #4 succeeded at 341. 1 out of 4 hunks failed--saving rejects to rc.php_ini_setup.rej done -rw-r--r-- 1 root wheel 1366 Nov 28 05:58 rc.php_ini_setup.patch
-rwxr-xr-x 1 root wheel 9395 Nov 28 05:58 rc.php_ini_setup.orig
-rwxr-xr-x 1 root wheel 9459 Nov 28 05:59 rc.php_ini_setup
-rw------- 1 root wheel 456 Nov 28 05:59 rc.php_ini_setup.rej/etc: cat rc.php_ini_setup.rej
@@ -265,10 +265,22 @@PHPFPMMAX=3
+PHPFPMIDLE=30
+PHPFPMSTART=1
+PHPFPMSPARE=2
+PHPFPMREQ=500
if [ $REALMEM -lt 250 ]; then
PHPFPMMAX=2
+ PHPFPMIDLE=5
+ PHPFPMSTART=1
+ PHPFPMSPARE=1
+ PHPFPMREQ=500
elif [ ${REALMEM} -gt 1000 ]; then
- PHPFPMMAX=4
+ PHPFPMMAX=8
+ PHPFPMIDLE=3600
+ PHPFPMSTART=2
+ PHPFPMSPARE=7
+ PHPFPMREQ=5000
fi/bin/cat > /usr/local/lib/php-fpm.conf < <eof<br>/etc: diff rc.php_ini_setup.orig rc.php_ini_setup
308c308
< pm.process_idle_timeout = 5
–-pm.process_idle_timeout = $PHPFPMIDLE
310c310
< pm.max_requests = 500
pm.max_requests = $PHPFPMREQ
318c318
< pm.process_idle_timeout = 5
pm.process_idle_timeout = $PHPFPMIDLE
320,321c320,321
< pm.start_servers = 1
< pm.max_requests = 500
pm.start_servers = $PHPFPMSTART
pm.max_requests = $PHPFPMREQ
323c323
< pm.max_spare_servers=1
pm.max_spare_servers= $PHPFPMSPARE
332c332
< pm.max_requests = 500
pm.max_requests = $PHPFPMREQ</eof<br>
After that I just restored the saved copy and just manually make the changes. Still working great. I'm using a PC engines apu2c2 and memory usage is about 12 percent Thanks again.
-
It's probably something i did wrong. Like I said I'm not familiar with BSD, and only used patch a couple of times.
Saved the patch listed in this thread as rc.php_ini_setup.patch
Use the System_Patches package, makes it a lot easier.
-
Ah, not familiar with the pfSense patch system. :)
OK, first thing to do is reverse what you have done, put everything back as it was.
Next, go into pfSense and select Package Manager from the System Menu, select 'Available Packages' and find the 'System Patches' package and install it.
You will now have a 'Patches' option in the system menu - Select it.
Click on add new patch.
In the URL/Commit ID enter the patch ID 028be76, give a description in the Description box and press save.
Now click on Test, it should say can be applied cleanly. Now click on Apply.
Job done.
-
Thank you again. I replaced the original file and re-applied the patch the proper way for pfSense. Other than some extra spaces I had it correct.
The router continues to be stable with this patch. -
That's good.
The advantage of using the patch system, apart from saving you a lot of work, is that you have a setting in the patch that allows for it to be automatically re-applied when you update pfSense. It checks to see whether it can be re-applied cleanly and does so. Obviously, if the file(s) the patches are applied against change in a way that means the patch will not cleanly apply then it does not do so, so when you use patches it's advisable to check after an update. For the average user who is not on the beta trail this means only a two or three times a year, beta testers should check every time they update.
-
tward800 thanks for your feedback, even if there is no problems, feedback is good, as it helps developers make a decision on if a patch is safe enough to rollout or not.
-
The patch gives me some errors when I test it:
/usr/bin/patch –directory=/ -t -p2 -i /var/patches/5a269040d9477.patch --check --forward --ignore-whitespace
Hmm... Looks like a unified diff to me...
The text leading up to this was:|From 2c131b10b25db593331048d4f2b28fbf9bf5662e Mon Sep 17 00:00:00 2001
|From: marjohn56
|Date: Wed, 29 Nov 2017 23:18:51 +0000
|Subject: [PATCH] Increase FPM process availability in high ram systems
|
|To reduce chance of nginx gateway error when interacting with FPM backend, this patch does the following, starts up extra FOM server processes at startup, allows more to stay running on standby, increases automatic shutdown time from 5 seconds to one hour. On systems with a gig or more of ram
|–-
| src/etc/rc.php_ini_setup | 29 +++++++++++++++++++++--------
| 1 file changed, 21 insertions(+), 8 deletions(-)
|
|diff --git a/src/etc/rc.php_ini_setup b/src/etc/rc.php_ini_setup
|index 5ce0ef7a31..03bc1178d9 100755
|--- a/src/etc/rc.php_ini_setup+++ b/src/etc/rc.php_ini_setup Patching file etc/rc.php_ini_setup using Plan A... Hunk #1 succeeded at 265. Hunk #2 failed at 317. Hunk #3 failed at 340. 2 out of 3 hunks failed while patching etc/rc.php_ini_setup done I'm using a new Pfsense 2.4.2 install and using the patch manager from the gui.
-
Try patch 2c131b1.
Yes, I had to re-sync with upstream. In doing so my Github desktop threw one and I had to pull it back and re-do the PR. However, it's against 2.4.3, so it may not work for 2.4.2 now.
