pfsense 2.6.0 sshguard @ web gui bug/crash

VioletDragon · Apr 17, 2022, 3:00 AM

@violetdragon I have noticed that PHP is using some RAM and some CPU when the Gui works.

23165 www           1  20    0    27M    16M kqread   1   0:59   0.11% haproxy
71979 root          1  20    0    13M  3644K CPU1     1   0:00   0.10% top
85706 root          1  27    0    60M    43M nanslp   1   0:31   0.09% php
23709 root          1  20    0    30M  9612K kqread   0   0:00   0.06% nginx
62037 root          3  41   20   538M   471M bpf      2   0:07   0.03% snort
68274 root          1  20    0    20M  9396K select   3   0:00   0.02% sshd
54051 dhcpd         1  20    0    25M    13M select   1   0:29   0.02% dhcpd
39229 root          5  52    0    11M  2644K uwait    0   0:36   0.01% dpinger
29141 root          1  20    0    21M  8408K select   1   0:04   0.01% mpd5
65272 root          1  20    0    11M  2200K select   3   0:53   0.01% powerd
95019 avahi         1  20    0    12M  3568K select   1   0:34   0.01% avahi-daemon
38649 root          5  52    0    11M  2644K uwait    1   0:42   0.01% dpinger
 2502 root          1  20    0    11M  2700K select   2   0:29   0.01% syslogd
31450 root          1  20    0    19M  7216K select   1   0:14   0.01% ntpd
77306 root          1  20    0    11M  2212K kqread   0   0:13   0.01% tail_pfb
85029 root          1  20    0    17M  7900K kqread   2   1:35   0.00% lighttpd_pfb
  372 root          1  20    0   101M    27M kqread   1   0:07   0.00% php-fpm
60982 root          1  20    0    12M  3024K bpf      3   0:48   0.00% filterlog
77510 root          1  21    0    77M    59M piperd   0   4:36   0.00% php_pfb
 5480 root          2  20    0    19M  7788K select   2   1:54   0.00% openvpn
12965 uucp          1  20    0    12M  2864K select   1   0:37   0.00% usbhid-ups
57324 root          1  52    0   134M    52M accept   3   0:19   0.00% php-fpm
58571 root          1  52    0   134M    52M accept   1   0:19   0.00% php-fpm
47138 root          1  20    0   132M    50M piperd   2   0:17   0.00% php-fpm
85596 root          1  52    0   132M    50M accept   0   0:16   0.00% php-fpm
 8628 root          1  52    0   134M    51M accept   0   0:15   0.00% php-fpm

VioletDragon · Apr 17, 2022, 3:18 AM

@violetdragon Just thought i'd mention this, after having another look it's something on the home page that is causing the issue, when the home page does not load if i go to any of the tab it loads them but not the home page.

(Edit)

The problem is the Disks Widget, when the Disk widget is on the home page the problem appears home page does not load when removed off the home page problem disappears

stephenw10 · Apr 17, 2022, 3:42 PM

Huh, that's interesting. The disks widget is there by default on on 2.6 installs so I would have expected many more reports of similar behaviour.
Do you have an unusual disk setup?
Is there anything logged in the nginx or system logs when this happens?

Steve

VioletDragon · Apr 17, 2022, 3:42 PM

@stephenw10 Hi, only disk setup I have are 2x 60GB Solid State Drives in a mirror, nope nothing in the logs, would it be possible to post a video so you can see it? It's strange ain't it.

stephenw10 · Apr 17, 2022, 8:37 PM

Sure post a video, or link to it. I'd like to see it.

I have systems with dual ZFS disks in a mirror but they are smaller.

Steve

jimp · Apr 19, 2022, 2:24 PM

@violetdragon said in pfsense 2.6.0 sshgaurd @ web gui bug/crash:

@stephenw10 Hi, only disk setup I have are 2x 60GB Solid State Drives in a mirror, nope nothing in the logs, would it be possible to post a video so you can see it? It's strange ain't it.

Is this a gmirror setup that's been upgraded over time or a ZFS mirror?

I have several ZFS mirrors and the disk widget works fine there but I don't think I have any gmirror setups on 2.6 currently.

stephenw10 · Apr 18, 2022, 1:42 PM

No problems on the test box I use for this:

Gertjan · Apr 19, 2022, 8:16 AM

Probably not related, but :
@violetdragon said in pfsense 2.6.0 sshguard @ web gui bug/crash:

2020/09/08 04:19:59 [error] 4127#100429: *20842 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.1.9, server: , request: "POST /acme/acme_certificates.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "violetdragon.ddns.net:10443", referrer: "https://violetdragon.ddns.net:10443/acme/acme_certificates.php"

Who is accessing what from where ?
Why is a LAN based client using "violetdragon.ddns.net" (the WAN IP ?? )- why not using the LAN IP of pfSense host name, which is 192.168.1.1 ?
Or is your pfsense really called "violetdragon" and your domain set to "ddns.net" ? So "violetdragon.ddns.net" is 192.168.1.1 (looks very wrong to me).

stephenw10 · Apr 19, 2022, 1:05 PM

It's unusual but it should work fine that way. The disks widget shouldn't care.

Gertjan · Apr 19, 2022, 1:05 PM

@stephenw10

Sure thing.
It looked to me as if the request came from the 'outside' which means he opened up the GUI to the outside world. And that opens up a can of worms.

VioletDragon · Apr 19, 2022, 2:22 PM

@gertjan If you look at the logs carefully, you will see that the 1.9 IP is my workstation, violetdragon.ddns.net was the DDNS Hostname of the firewall and I was internally wrapping it inside meaning, I was using the DDNS Hostname with DNS Resolver it is not unusual to do, I moved to two Static IPs for Ha on my WAN so now i am using a proper FQDN with DNS Resolver & Haproxy with SSL Offloading for Lets Encrypts for both Internal Services and External Services, I guess your not familiar with this kind of setup, and yes I have moved the IP of the Firewall from 1.1 this is what you do in the CCNA world. Web Gui is not publicly exposed I am not that dumb to publicly expose the Web Gui same with SSH on everything, for External use I use my FQDN and OpenVPN/IPsec for offsite Servers.

VioletDragon · Apr 19, 2022, 2:24 PM

@jimp Hi, it is a ZFS Mirror.

stephenw10 · Apr 19, 2022, 2:30 PM

Mmm, not seeing any issues on systems with ZFS mirrors here.
Hopefully the video should clarify things.

Steve

VioletDragon · Apr 19, 2022, 2:36 PM

@stephenw10 I will get the video to you in a few hours, I have had a busy weekend with it being bank holiday. Sorry for the delays.

stephenw10 · Apr 19, 2022, 2:47 PM

No worries, I'm glad you were able to narrow down the cause this far already.

VioletDragon · Apr 20, 2022, 1:00 AM

@stephenw10 Hi, Just to report back. Even after removing the Disk Widget the problem is still there cannot access the web gui at all then it starts working but can access different tabs. This is weird.

VioletDragon · Apr 20, 2022, 1:50 AM

@stephenw10 Hi, Here is the Video, I had to put it on one of my Servers, Hopefully you will be able to play it. This is from one of my Firewalls in a different location, I have another with the same issue as well. (https://cloud.violetdragonscloudnetwork.co.uk/s/iCKFFgmqQ8jLQ5a)

Gertjan · Apr 20, 2022, 7:46 AM

@violetdragon

What happens when you use the 'admin' user ?
Jack is fine, but that's probably not an 'admin'.
Dome info collected by the dashboard GUI page need 'admin' rights.

Why creating a user like Jack ? pfSense is a router firewall, not some file server.

VioletDragon · Apr 20, 2022, 8:02 AM

@gertjan Security, admin is a easy username to guess. Admin user does the exact same. But I have a ton of PHP-FPM processes when this happens. I've also triggered the problem by attempting brute-forces via SSH. I have va feeling that maybe something is going on.

Gertjan · Apr 20, 2022, 8:20 AM

@violetdragon said in pfsense 2.6.0 sshguard @ web gui bug/crash:

Security, admin is a easy username to guess.

Yeah, but who cares ?
Normally, the GUI should only be accessible from the LAN interface.
The other LAN ( OPT1 OPT2 OPT3 ) interfaces are meant to be used for you local network.
This allows you to even completely disconnect the LAN interface when you don't need the GUI access. That what security is.

Easy user names and passwords are a thing on a public network.
Your LAN is not a public network.

I've changed a user called '001' so it has admin privilges, like this :

Now I can login with '001' and the dashboard shows up in half a second.

What happens when you remove all or most of the the widgets ?
Login again to try again.