After upgrade 502 Bad Gateway

hudriwudri5 · Dec 18, 2017, 5:04 PM

Hi everybody,

I've upgraded today to succested new Version.
After reboot I was not able to open the Login page. I always get "502 Bad Gateway"

After looking for a resolution I've found the hint to restart PHP-FPM - so I logged in via SSH and did it.
For a short time I can Login and go to Settings. - then I get "504 Gateway Time-out".

The Version shown on Dashboard is
2.4.2-RELEASE-p1 (amd64)
built on Tue Dec 12 13:45:26 CST 2017
FreeBSD 11.1-RELEASE-p6

Routing and DHCP works.

Also I disabled pfBlockerNG - as mentioned in some articles - for example here: https://got-tty.org/pfsense-2-4-bad-Gateway
… but it doesen't help me.

Thank you in advance for help!

Gerhard

shortspecialbus · Dec 19, 2017, 12:04 AM

I'm having the same issue after upgrading today. I can reset PHP-FPM via ssh as a workaround but it isn't ideal.

Edit: For a bit more info, nginx is losing its socket file, I'm unclear as to why at this time. I'm digging a bit more.

Guest · Dec 19, 2017, 11:23 AM

There are a couple of patches you might want to try that are aimed at fixing this issue.

The first is actually included in the latest version of pfBocker so I guess that one isn't helping, so try this one: 2c131b1

That patch was created against 2.4.3, hopefully it will apply.

hudriwudri5 · Dec 19, 2017, 5:18 PM

Thank you for the answer.

But I'm a real beginner in terms of patching. I am a real pfSense DAU (german synonym for aprox. BDU).
Maybe there is anybody out there you can tell me how to patch this with "2c131b1"?

thank you in advance
Gerhard

Guest · Dec 19, 2017, 5:49 PM

First thing to do is to go to system->Package manager and install the system_patches pkg.

You will then have an system->Patches

Click Add New Patch

Give the description some title… 502 fix or something like that.

Enter the ID I gave you earler in the URL/Commit ID

Click Save.

Now, when the page returns you will see the option to Apply Test or Revert, click Test to see if it can apply cleanly. If it says yes then Click Apply. That's it, you can remove the patch by clicking revert.

Hope that helps.

hudriwudri5 · Dec 19, 2017, 6:02 PM

Thank's this helps me!

…. but unfortunately I can't apply
Patch can NOT be applied cleanly (detail)
Patch can NOT be reverted cleanly (detail)

hmmm :-(

Do you think it helps to reinstall it completle new?

Guest · Dec 19, 2017, 8:29 PM

No, see my comment about 2.4.3, you need to switch your repo to the development branch and update to 2.4.3. it's very stable, so there are no issues.

select update, select branch and select the development branch, update and see how you get on then.

ivor · Dec 19, 2017, 8:42 PM

marjohn56 - unless I'm wrong the issue is fixed in the latest pfBlockerNG pacakge. There's no need to add any custom patches.

Guest · Dec 19, 2017, 8:43 PM

He said in an earlier message the removing pfblocker did not solve the issue

ivor · Dec 19, 2017, 8:43 PM

hudriwudri5 - please try a clean re-install. Latest pfBlockerNG package update should not cause any 502 errors.

ivor · Dec 19, 2017, 8:44 PM

@marjohn56:

He said in an earlier message the removing pfblocker did not solve the issue

Some setting might be left. It also might not be related to pfBlockerNG but some other issue, update gone bad, but since we have little or no information, clean re-install is the best option.

Guest · Dec 19, 2017, 8:49 PM

There were 502 errors that were not pfblocker related, but you know best. I'll stay out of it and let you sort the issue out.

Bye,,, and thanks for all the fish.

lasocean · Dec 20, 2017, 3:25 PM

I am also seing this issue after updating to 2.4.2_1 (from 2.4.2)

Only thing that helps is rebooting. Then issue recurs after a few minutes.

Restarting PHP-PFM does not help.
Starting php-pfm indicates 'Bus error (core dumped)
Re-executing Restart PHP-PFM then indicates that pidfile cannon be opened when attempting to kill php-pfm, since php-pfm didn't restart.

After Reboot firewall will indicate that there was crash.
Crash report was submitted.

Initially, this appeared to also affect OpenVPN Connections, but after reboot OpenVPN connections have remained online, though web interface is inaccessible.

hudriwudri5 · Dec 20, 2017, 6:40 PM

I will try it in this order:
1.) reinstall pfBlockerNG
2.) (I've set Keep settings) therefore if it will not work I will deinstall pfBlockerNG without keeping Settings and reinstall it
3.) update to developer branch
4.) clean reinstall

@lasocean:
For me it works with restart PHP-FPM
Then I can click exaktly one time a link - the seconds one hangs. I restart PHP-FPM and I can go forward - step by step ;-)

I will post the result of my testing

Thank's a lot

edit:
After Step 1 now it seems to work!!!

Thank you all!

lasocean · Dec 20, 2017, 6:10 PM

Just a note…
I am not using pfBlockerNG addon, so that shouldn't be an issue.

No problem until actual login to Web Interface.
Can refresh login page any number of times without actually logging in.
Gonna see how far I can get after logging in a bit.

shortspecialbus · Dec 20, 2017, 11:48 PM

I don't have pfblocker myself. My 502 errors seem to start after trying to restart a service, such as bandwidthd (I was trying to get Postgres working before I realized that postgres support isn't available in pfsense's bandwidthd presently.)

I'll look into applying that patch if I continue having issues, so far it's been stable for a few days, but possibly because I've only been looking at things rather than restarting services

lasocean · Dec 21, 2017, 12:06 AM

I don't know if it's related, but…
I decided not to login to the WebConfigurator until after most users had left for the day.
Immediately after logging in, I removed OpenVPN Widget from the Dashboard, just in case it's something on the dashboard that causing an issue.
I believe that's the only widget I had added to the Dashboard. But, it's not a recent addition.
I have been able to view different pages without the running into Bad Gateway issue, so far.

All this may be irrelevant.
I will post an update tomorrow re recurrence of the issue.

We have had a couple glitches with provider for WAN link in the last couple days. Dunno if WAN going down or coming back online triggered issue.

lasocean · Dec 22, 2017, 8:23 PM

Update
Sometimes receive Bad Gateway Error, but simply refreshing the page will have it load correctly.
If Bad Gateway persists after multiple Refresh attempts, restarting PHP-PFM seems to help.

Browsing and OpenVPN Connections seem to be having difficulty at times.
Sometimes restarting the system helps for a while.

Only started seeing these issues since Patch 1 Update for 2.4.2

chrcoluk · Dec 23, 2017, 7:25 AM

https://forum.pfsense.org/index.php?topic=110515.msg766964#msg766964

More information on what me and Martin worked on, I am hoping my patch which Martin submitted on github got accepted, but maybe it didnt, as the patch should resolve this issue for anyone using pfsense on hardware with at least 1 gig of ram.

–edit--

It has been approved 2 days ago.

https://github.com/pfsense/pfsense/pull/3881
https://redmine.pfsense.org/issues/8125