502 bad gateway

oasis_ck

Hi, first commit here.
Once a day, pfSense crash with error 502 bad gateway. Got massive subjects with googling that. None of answer work. You're my last chance.
My setup :

pfSense 2.4.3-RELEASE-p1
i3-4030
4GB RAM
4 NIC (one wan, one lan with wifi router, 2 disabled)

What I use/install from a fresh install (dit it 3-4 times) :

DHCP + DNS
OpenVPN + openvpn client export pkg
Acme pkg
SNMP
dpinger disable
Change the binary dhclient for the WAN (https://lafibre.info/remplacer-livebox/probleme-pour-remplacer-une-livebox-4-par-un-routeur-pfsense/)

What I do when it crash :

SSH + option 16 => not working
reboot => working

Thanks

oasis_ck

this is my system.log while it's crashing

Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
[...]
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket
Sep  3 19:13:44 pfSense check_reload_status: Could not connect to /var/run/php-fpm.socket

x 5000
I saved log directory so if you need more log tell me. But please help I'm loosing trust about pfSense...

oasis_ck

this morning

Sep 4 05:01:03	check_reload_status		rc.newwanip starting igb0.832
Sep 4 05:00:50	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:50	check_reload_status		Linkup starting igb0
Sep 4 05:00:50	kernel		igb0.832: link state changed to UP
Sep 4 05:00:50	kernel		igb0: link state changed to UP
Sep 4 05:00:45	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:45	check_reload_status		Linkup starting igb0
Sep 4 05:00:45	kernel		igb0.832: link state changed to DOWN
Sep 4 05:00:45	kernel		igb0: link state changed to DOWN
Sep 4 05:00:42	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:42	check_reload_status		Linkup starting igb0
Sep 4 05:00:42	kernel		igb0.832: link state changed to UP
Sep 4 05:00:42	kernel		igb0: link state changed to UP
Sep 4 05:00:38	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:38	check_reload_status		Linkup starting igb0
Sep 4 05:00:38	kernel		igb0.832: link state changed to DOWN
Sep 4 05:00:38	kernel		igb0: link state changed to DOWN
Sep 4 05:00:38	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:38	check_reload_status		Linkup starting igb0
Sep 4 05:00:38	kernel		igb0.832: link state changed to UP
Sep 4 05:00:38	kernel		igb0: link state changed to UP
Sep 4 05:00:34	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:34	check_reload_status		Linkup starting igb0
Sep 4 05:00:34	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:34	check_reload_status		Linkup starting igb0
Sep 4 05:00:34	kernel		igb0.832: link state changed to DOWN
Sep 4 05:00:34	kernel		igb0: link state changed to DOWN
Sep 4 05:00:34	kernel		igb0.832: link state changed to UP
Sep 4 05:00:34	kernel		igb0: link state changed to UP
Sep 4 05:00:30	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:30	check_reload_status		Linkup starting igb0
Sep 4 05:00:30	kernel		igb0.832: link state changed to DOWN
Sep 4 05:00:30	kernel		igb0: link state changed to DOWN
Sep 4 05:00:24	check_reload_status		Linkup starting igb0.832
Sep 4 05:00:24	check_reload_status		Linkup starting igb0

I have fiber and got no prob with ISP box. I'm using same cable.
I had to reload PHP-FPM.

oasis_ck

New update.
I captured in live the up and down of pfSense.

clog -f /var/log/dhcpd.log > dhcpd.log
clog -f /var/log/system.log > system.log
0_1536087081944_log.zip

The only way to have a decent WAN for hours is hard reboot.

Thanks

oasis_ck

I switch to OPNsense because dhclient is native for my configuration : https://docs.opnsense.org/manual/how-tos/orange_fr_fttp.html
No down interface since 1 week, the logs are clean.

Gertjan

Hi,

@oasis_ck said in 502 bad gateway:

Change the binary dhclient for the WAN (https://lafibre.info/remplacer-livebox/probleme-pour-remplacer-une-livebox-4-par-un-routeur-pfsense/)

This link talks about modifying programs. What did you download ?
These programs would run on FreeBSD 10 ? 32 bits ?
Now you are using 64 bits - FreeBSD 11.x

What I want to say : you can't mix 32 and 64 bits programs, neither when the are compliked against another OS (OS libs/headers).

Did you take all this in account ?

oasis_ck

Hi @Gertjan , thanks for your interest
Yep I take all this in account and, by the way, I also tried with dhclient binary from OPNsense to pfsense and it's exactly the same behavior.
Finally, OPNsense is like pfsense for my issue...
I plug the ISP box on the WAN interface (OPNsense now) and no log out since 2 weeks.
But I don't like that (more wires, more stuff) and I really want to use pfsense/opnsense directly to WAN.

I restart from the beginning :
now, if I use modifying binary or opnsense native binary, I have a WAN access with random log out and 2 erros in my logs when I catch an IP :

Sep 22 12:21:50 OPNsense dhclient[27297]: unknown dhcp option value 0x5a
Sep 22 12:21:50 OPNsense dhclient[27297]: unknown dhcp option value 0x78

What thoses errors are related to dhcp option ?

Gertjan

@oasis_ck said in 502 bad gateway:

Hi @Gertjan , thanks for your interest
Yep I take all this in account and, by the way, I also tried with dhclient binary from OPNsense to pfsense and it's exactly the same behavior.
Finally, OPNsense is like pfsense for my issue...
I plug the ISP box on the WAN interface (OPNsense now) and no log out since 2 weeks.
But I don't like that (more wires, more stuff) and I really want to use pfsense/opnsense directly to WAN.

I restart from the beginning :
now, if I use modifying binary or opnsense native binary, I have a WAN access with random log out and 2 erros in my logs when I catch an IP :
Sep 22 12:21:50 OPNsense dhclient[27297]: unknown dhcp option value 0x5a
Sep 22 12:21:50 OPNsense dhclient[27297]: unknown dhcp option value 0x78
What thoses errors are related to dhcp option ?

I tried this : https://www.google.com/search?q=DHCP+option+0x5a&ie=utf-8&oe=utf-8&client=firefox-b

Le "0x5a" : https://lafibre.info/remplacer-livebox/cacking-nouveau-systeme-de-generation-de-loption-90-dhcp/48/

Etc.

What I make from it :
France ;)
ISP == Orange.
You are trying to replace your Livebox ....
(am I still right ? ;))

Their are many forums (France) with threads that discuss the subject : ditch the Livebox ^^

oasis_ck

Yeah, you're right
I followed many thread before posting here. Most of thread are dated from 2016 and some guys running my issue. They resolved it by repluging this f#!* livebox and maybe the others don't care about random disconnection.

I'm really motivate to resolve this. I attach my full dhclient log (clog /var/log/dhcpd.log | grep dhclient). As you can see, the longuest disconnection is 26 sept. I replugged my box in the evening.
dhclient.zip