502 Bad Gateway (nginx) after Update to 2.3

tonymorella

Adding to the pain. Tonight my cable modem went up and down a few times, and pfsense went nutty :) I have gateway monitoring disabled on all interfaces. In the logs I see

The link go down
check_reload_status kick off
Reloading filter
link come back up
xinet Starting reconfiguration
rc.newwanip

Then and error:


Oct  5 01:45:22 pfSense php-cgi: rc.banner: PHP ERROR: Type: 1, File: /etc/inc/interfaces.inc, Line: 80, Message: Call to undefined function pfSense_interface_listget()

The cable modem sets a default IP 192.168.100.20 which kicked off check_reload_status: Reloading filter
xinet Starting reconfiguration
rc.newwanip which in turn kicking off
Dynamic DNS and OpenVPN that errored out because it did not have a public IP yet.
WAN receives a public IP and the services start, if I wait the GUI will come back, or restarting the web and php-fpm services via the SSH menu options 11 and 16

The "Starting reconfiguration process" happened 16 between 1:44:58 and 01:58:20, then it received a public IP. During the cycle check_reload_status was at 100% CPU on a PCEngines APU2 Quad Core, I could not get in via GUI only SSH. The above error happened 16 times, each time it cycled.

Unless I missed an option did not see a way to delay the Starting reconfiguration process, looking at the source to see what I can find.

Comments?

Tony

luckman212

Came here to cry about the same problem. 2.3.2_1. I have the OpenVPN widget on my dash but not the IPSEC widget. Is there something I can patch manually to stop this for now?

edmund

My solution has been to disable auto-negotiation on the WAN interface and fix the pfSense WAN interface at 100baseTX full-duplex - the has completely solved the issues for my home setup since my cable connection is only 65/10 Mb on a good day.

It's my suspicion that the issues are caused by auto-negotiate failing in some subtle way - however I'm running pfSense on a Chinese made board so I can't be certain the the actual NIC is really made by Intel as it claims. I'm open-minded about this - I don't see these problems on another pfSense box at work in an identical WAN configuration on an SG-4860.

luckman212

Hmm I don't know about that- i'm having the issue on an intel (sg2440). Can't set to 100fdx because the WAN is 300mbit cable.

edmund

You could try just setting the link speed instead of auto-negotiating it - it was my suspicion that auto-negotiate was failing.

khaled

please choose 16
Restart php-fpm
and try agine

igpit

I just experienced this with 2.3.3-RELEASE-p1 !

Today when I want to check the admin web page I get the 502 error.

Running option 16 from console solved the issue. I thought this was fixed by now?

igpit

It just happened again. "Restart php-fpm" solved it, but there is definitely some bug.

weehooey

Have same issue on 2.3.4
Restarted PHP-FPM restored GUI and OpenVPN
Removed IPsec widget from dashboard, hopefully will help

AlexMex

Hello,

I'm getting the 502 bad gateway too. I have just installed pfsense 2.3.4.
I start getting the issue after setting up four VLANs on my OPT1 interface. Using option 16 Restart PHP-FPM is working sometimes on first shot but more frequently after second or third attempt.
CPU usage was arround 1.5%

Today I install SQUID and activate transparent proxy mode on my LAN and four OPT1 VLANs.
After I login in webconfigurator, I cannot access any page. I immediately fall on the nginx 502 bad gateway error. I have tried to use option 16 as before but it does not work anymore. Only a reboot of pfSense box allow me to login again in the web configurator.

After several reboots I saw that sometimes the OPT1 appears down and the VLANs are up, , sometimes OPT1 and VLANs are down, sometimes everything is up as expected :-[
Each time CPU usage increase to 100% and then in a few seconds I got the 502 message.

I have made some tests and when I disconnect cable on OPT1, issue does not happen.
I have checked option "Do not forward traffic to Private Address Space (RFC 1918) destinations" and plug in the cable on opt1. It lloks like I am back to initial situation now.

It fails less often but is still annoying since I have to reboot the box to continue my configuration.

If you have any suggestion with this I will really apreciate. For now, I will stay on pfSense 2.2

costasppc

Ι can confirm the issue with 2.3.4-RELEASE (amd64). When the gui is not accessible, OpenVPN users cannot login. When I used option 16, the users could login again.

I disabled the widgets mentioned before (although I need the OpenVPN widget…) and see what's happen.

Best regards

Kostas

edmund

@edmund:

I believe that the root of all my problems has been an auto-negotiate failure on the WAN interface …

Update: I'm convinced that my problems all stem from using a cheap Chinese "pfSense" system that I purchased on Amazon - it was about half the price of a comparable Netgate unit but it has continuously generated errors on the WAN interface and has never managed to auto-negotiate the link speed. Recently I had the cable company at the house after complaining that the cable speed (at 10Mbs) was too low - I'm paying for 150Mbs. After about an hour of trying everything and failing to fix the problem I put a switch in between the cable modem and the firewall wan port - which boosted the speed to about 55Mbs.

I have replaced the "el cheapo" Chinese box with a Netgate box - auto-negotiated works, the cable modem instantly supports 1Gbs and I'm not getting 160Mbs on the cable connection. I'm no longer seeing any errors on the WAN interface.

hdejongh

sunday i updated 6 firewalls from 2.3 to 2.4.
All suddenly are showing 502 bad gateway problems.
Besides that 2 of them become unresponsive after around 20 hours.
Its hard to access the firewall then but the one time i got lucky i can see that the memory is completely full.
So i doubled the memory and still same problem occur.

Only way to solve it is rebooting the firewall

btw, all are vm's
2 of them were a complete new install with a config restore (i had to go from 32bits to 64 bits).

My physical pfsense's are not infected…

jahonix

Search in this forum or on redmine.pfsense.org - this was seen before.
IIRC it may come from an associated CD/DVD drive to your pfSense VM. Get rid of that and it might work. Other problem I don't remember off of my head is discussed on redmine.

rightnow

@hdejongh:

sunday i updated 6 firewalls from 2.3 to 2.4.
All suddenly are showing 502 bad gateway problems.
Besides that 2 of them become unresponsive after around 20 hours.
Its hard to access the firewall then but the one time i got lucky i can see that the memory is completely full.
So i doubled the memory and still same problem occur.

Only way to solve it is rebooting the firewall

btw, all are vm's
2 of them were a complete new install with a config restore (i had to go from 32bits to 64 bits).

My physical pfsense's are not infected…

Exactly same problem here! Firewall dies, and 502 bad gateway.

mikael.andre

Hello,

I experienced the same problem on 2.4.0 release.
I have no access to my firewall by :

SSH
WebGUI
Console

BUT, I still continue to surf on the Internet… Very strange...
This issue occurs at the end of 20 hours uptime.
It's hardening to identify the root cause.
The only way to resolve this problem is rebooting my hardware appliance.
Once the reboot process is done, there is no event logs...

I have the following widget on my dashboard :

System Informations
NTP Status
SMART Status
pfBlockerNG
OpenVPN
Gateways
Interfaces
Interfaces Statistics
Traffic Graphs
Services status
Firewall logs
Thermal sensors

Heres my hardware configuration :

MotherBoard : APU2C4
CPU : AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support
RAM : 4 GByte DDR3-1333 DRAM
Storage Type : mSATA SSD
Storage Size : 120GB
Ethernet ports : 3 x 1Gbit/s
Wireless card : WLE200NX with two antenna

BBcan177

Update pfBlockerNG to v2.1.2 and reboot.

https://forum.pfsense.org/index.php?topic=137103.0

mikael.andre

Hi BBcan177,

Many thanks.
I was already in this version. But, I deleted some widget and increased the number of webconfigurator process up to 4.
I also disabled autocheck updates in Dashboard and deinstalled NTOPNG due to some unstabilities.
I keep an eye on it.

pdrass

This has reared its ugly head again on the latest 2.4.1 release. I have about 10% of my FW's doing this.

I tried to update the packages:

pkg update
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.

On the webUI it sits and spins.

So…apparently the repo's are really down and you can't get to them?

https://forum.pfsense.org/index.php?topic=139276.0

I updated two of the bad apples today in hopes they'd be fixed form the nginx problem.

I can't ssh to it, when I login it just says "pfsense -" <-- I can't do the /etc/init....whatever to start the console number list, I can't shutdown -r now, nothing...it just sites there. I also can't OpenVPN into the box which is when I start getting the phone calls then someone has to go in and reboot the firewall.

Is this PFBng doing this? Perish the thought! :0)

1 - When will the repos be back up?
2 - I swear I remembered to update the packages on each box after updating but maybe I missed these two

Once again...thanks PFSense for all you do, the ONLY firewall I use and deploy in production!

BBcan177

Try to hit: CTRL - z and then enter "/bin/tcsh"