GUI Faling to respond (Developer comment requested)

guardian · Jul 13, 2021, 8:13 PM

I had an ISP failure where the link has gone down, and when I attempted to log into pfSense, the WebGUI was non-responsive.

I'm not sure why this occurred, I would assume the CPU was so busy the GUI could not run (Am I correct that the GUI runs a lower priority). The shell was quite responsive, but other than killing PHP-FPM, I wasn't really sure what to do. After killing PHP-FPM, I was able to log in.

I've added a bit more info at the end of this post with info about the log entries most likely of significance. I can understand why these messages sh/would be being generated, but I'm wondering if some sort of throttling/rate limiting should be added or if I need to make some sort of configuration change.

Hopefully one of the devs will see this and can comment if this gives rise to an possible improvement. It's not great when an event with the ISP causes a pfSense failure.

Additional Info:

I checked the log, and was getting one or two of these per second:
May 8 00:15:29 XXXXXXXX kernel: arpresolve: can't allocate llinfo for xxx.xxx.xx.1 on em0

When I power cycled the modem I started getting:
May 8 00:16:08 XXXXXXXX kernel: arpresolve: can't allocate llinfo for 192.168.100.1 on em0

I also have an OpenVPN Server that is pushing out quite a few messages as it is trying to get reconnected.

I also got a couple of these
May 8 00:11:34 XXXXXXXX php-fpm[80642]: /rc.linkup: The command '/usr/sbin/arp -s 'LANVPNIP.1' '00:28:1a:e0:10:05'' returned exit code '1', the output was 'arp: writing to routing socket: Operation not permitted'

Gertjan · Jul 14, 2021, 1:27 PM

Hi,

@guardian said in GUI Faling to respond (Developer comment requested):

I had an ISP failure where the link has gone down

What 'link' ?
On the ISP side of the modem, the coax or POTS(ADSL) connection ?
Or the NIC side (Ethernet) connection of the modem ?
Both ?

If the 'WAN' side of pfSense goes down, a view messages are logged, abd as soon as it comes up, far more messages pop up, as many processes get restarted, as probably the WAN IP, an RFC1918 IP or a real Internet IP (your case I guess) gets initialized.

About the GUI access : when you login, or go to the 'Dashboard page' a lot of info is collected to be shown on that page. Some of the info is collected from locations that are not local. It comes from 'somewhere' on the Internet. Typically : servers hosted by Netgate. These are not addressed by IP, but FQDN, so they have to be resolved first. This needs an Internet connection.
Most - if not all - DNS interactions will time out after a minute (more ?) or so, and the Dashboard should show up.

Test : set up a short cut in your browser to a page like
https://yourpfsense.yournetwork.tld/status_services.php

Close the browser.
Remove the pfSense WAN cable.
Open the browser, and go to the short cut.
You have to login, that page shows up right away.
After entering login credentials, you will see the "services" page immediately - no delay.
When you go from here to the Dashboard page, you probably still have to wait a bit.

Btw : the additional info you posted are the classic :
"Hey, some one ripped out the WAN cable"
notifications.
These are harmless and as you have seen yourself : rather rare.

May 8 00:15:2 ....

that was more then two months ago ;)

@guardian said in GUI Faling to respond (Developer comment requested):

I also have an OpenVPN Server that is pushing out quite a few messages as it is trying to get reconnected.

It's just one of the process that was listening (== bound to) the WAN NIC, and now the openvpn process was kicked in the face.
When it comes back - it restarted ? it attaches itself to the WAN interface again, and starts listening.
Be aware : the OpenVN server is a server type of application, it doesn't connect, it just listening..

guardian · Jul 14, 2021, 7:17 PM

@gertjan said in GUI Faling to respond (Developer comment requested):

Hi,

@guardian said in GUI Faling to respond (Developer comment requested):
I had an ISP failure where the link has gone down
What 'link' ?
On the ISP side of the modem, the coax or POTS(ADSL) connection ?
Or the NIC side (Ethernet) connection of the modem ?
Both ?

Cable modem connected to pfSense with a CAT5 Cable. The cable modem went off line (likely due to maintenance in the area). The status lights indicated that the modem had lost connectivity.

If the 'WAN' side of pfSense goes down, a view messages are logged, abd as soon as it comes up, far more messages pop up, as many processes get restarted, as probably the WAN IP, an RFC1918 IP or a real Internet IP (your case I guess) gets initialized.

About the GUI access : when you login, or go to the 'Dashboard page' a lot of info is collected to be shown on that page. Some of the info is collected from locations that are not local. It comes from 'somewhere' on the Internet. Typically : servers hosted by Netgate. These are not addressed by IP, but FQDN, so they have to be resolved first. This needs an Internet connection.

Good point.... is there some way I should configure the system to mitigate this?

In this case I wasn't even getting the login page.

Most - if not all - DNS interactions will time out after a minute (more ?) or so, and the Dashboard should show up.

Test : set up a short cut in your browser to a page like
https://yourpfsense.yournetwork.tld/status_services.php

What does this do? Can I test with it when the system is working?

I need a bit of help interpreting yourpfsense.yournetwork.tld. Is this the routername.xxx in the login that I normally use to login to the router: https://routername.xxx/? Being a residential cable customer I have a reverse dns something like CPExxxxxxxxxxxxxxxxxxxx.rogers.com. Does this mean more literally yourpfsense.CPExxxxxxxxxxxxxxxxxxxx.rogers.com?

Close the browser.
Remove the pfSense WAN cable.
Open the browser, and go to the short cut.
You have to login, that page shows up right away.
After entering login credentials, you will see the "services" page immediately - no delay.
When you go from here to the Dashboard page, you probably still have to wait a bit.

Btw : the additional info you posted are the classic :
"Hey, some one ripped out the WAN cable"
notifications.
These are harmless and as you have seen yourself : rather rare.
May 8 00:15:2 ....
that was more then two months ago ;)

@guardian said in GUI Faling to respond (Developer comment requested):

I also have an OpenVPN Server that is pushing out quite a few messages as it is trying to get reconnected.

It's just one of the process that was listening (== bound to) the WAN NIC, and now the openvpn process was kicked in the face.
When it comes back - it restarted ? it attaches itself to the WAN interface again, and starts listening.
Be aware : the OpenVN server is a server type of application, it doesn't connect, it just listening..

AFAIK things come back OK... I've had 3 outages in the last day or so. I had problems with the OpenVPN Client losing connectivity, so I built a small watchdog script that restarts the client after 5 minutes.

mccann25 · Jul 16, 2021, 7:49 AM

Hi, I have just encountered this issue and was going to log a ticket but will report here first.

I am running 2.5.2-RELEASE, I had an ISP outage today (rare) and while was trying to diagnose the issue I found Pfsense web interface in a hung state, after some investigation and a reboot, I found I could login again, but the GUI will hang after 2-5mins. I tried rebooting with the WAN interface disconnected (unplugged the cable) and the GUI remained responsive, the issue only seems to occur when the WAN interface cant pick up an ip from my bridged modem. Once the GUI has hung, the browser returns a bad gateway error.

login-to-view

Gertjan · Jul 16, 2021, 8:21 AM

@guardian said in GUI Faling to respond (Developer comment requested):

I need a bit of help interpreting yourpfsense.yournetwork.tld

I mean : whatever you've entered here :

login-to-view

you can use this fqdn to address yourself to your pfSense.

Or : use the classic "192.168.1.1"

Gertjan · Jul 16, 2021, 8:27 AM

login-to-view

127.0.0.1 will do. Why all the others ?

There is a time out for all those external DNS servers, each of them, which are not reachable, as your WAN is down.
127.0.0.1 is reachable, as it is local.
Yet another reason not to add any DNS servers.

Btw : the "bad gateway" message means : the web server became impatient, as the underlying PHP interpreter was still occupied executing code, waiting for a link coming up to one of the DNS servers, or some other external site like the pfSense file update servers.

guardian · Jul 19, 2021, 8:30 PM

@gertjan said in GUI Faling to respond (Developer comment requested):

[SNIP]
About the GUI access : when you login, or go to the 'Dashboard page' a lot of info is collected to be shown on that page. Some of the info is collected from locations that are not local. It comes from 'somewhere' on the Internet. Typically : servers hosted by Netgate. These are not addressed by IP, but FQDN, so they have to be resolved first. This needs an Internet connection.
Most - if not all - DNS interactions will time out after a minute (more ?) or so, and the Dashboard should show up.

Test : set up a short cut in your browser to a page like
https://yourpfsense.yournetwork.tld/status_services.php

Close the browser.
Remove the pfSense WAN cable.
Open the browser, and go to the short cut.
You have to login, that page shows up right away.
After entering login credentials, you will see the "services" page immediately - no delay.
When you go from here to the Dashboard page, you probably still have to wait a bit.

@gertjan said in [GUI Faling to respond (Developer comment

I mean : whatever you've entered here :

login-to-view

you can use this fqdn to address yourself to your pfSense.

Or : use the classic "192.168.1.1"

In my case I use a different IP address, but IIUC this is essentially the normal login address with /status_services.php appended to the end.
I tried logging out, closing the browser (and clearing the cache) and entering http://xx.xx.xx.xx//status_services.php, and indeed I did get the logon screen as usual, but I did not see the Services page, I got the dashboard once I entered by credentials. Does this only work if the WAN link is disabled?

@gertjan said in GUI Faling to respond (Developer comment requested):

login-to-view

127.0.0.1 will do. Why all the others ?

There is a time out for all those external DNS servers, each of them, which are not reachable, as your WAN is down.
127.0.0.1 is reachable, as it is local.
Yet another reason not to add any DNS servers.

Btw : the "bad gateway" message means : the web server became impatient, as the underlying PHP interpreter was still occupied executing code, waiting for a link coming up to one of the DNS servers, or some other external site like the pfSense file update servers.

The input from @mccann25 got me thinking. I'll bet it is a DNS issue... no internet = no dns => One very confused/overloaded/blocked DNS resolver??

During the outage when I used http://fqdn things seemed to hang and I didn't get a login screen- maybe I didn't wait long enough for things to time out. Using the IP address at least got a login screen after a few seconds, but then I didn't get a dashboard. (That leads me to the conclusion that the DNS Resolver was not responding -- either hung or blocked. Does this point to a reconfiguration on my part, or is it to be expected that local DNS on a LAN VPN would not be responding? It should not be necessary to go to the WAN to resolve the pfSense host name.)

I'm wondering how many of those DNS requests were in parallel vs serial? DNS--TIMEOUT--DNS--TIMEOUT--DNS--TIMEOUT--DASHBOARD. If the timeout is 30s, that means waiting 1.5 minutes for a dashboard.

Any suggestions/comments?

guardian · Jul 19, 2021, 8:33 PM

@mccann25 I think we are having the same problem. I only have 1 DNS Resolver 127.0.0.1 -- I didn't get the Bad Gateway message, but I likely didn't wait long enough.