pfsense 2.6.0 sshguard @ web gui bug/crash
-
Sure thing.
It looked to me as if the request came from the 'outside' which means he opened up the GUI to the outside world. And that opens up a can of worms. -
@gertjan If you look at the logs carefully, you will see that the 1.9 IP is my workstation, violetdragon.ddns.net was the DDNS Hostname of the firewall and I was internally wrapping it inside meaning, I was using the DDNS Hostname with DNS Resolver it is not unusual to do, I moved to two Static IPs for Ha on my WAN so now i am using a proper FQDN with DNS Resolver & Haproxy with SSL Offloading for Lets Encrypts for both Internal Services and External Services, I guess your not familiar with this kind of setup, and yes I have moved the IP of the Firewall from 1.1 this is what you do in the CCNA world. Web Gui is not publicly exposed I am not that dumb to publicly expose the Web Gui same with SSH on everything, for External use I use my FQDN and OpenVPN/IPsec for offsite Servers.
-
@jimp Hi, it is a ZFS Mirror.
-
Mmm, not seeing any issues on systems with ZFS mirrors here.
Hopefully the video should clarify things.Steve
-
@stephenw10 I will get the video to you in a few hours, I have had a busy weekend with it being bank holiday. Sorry for the delays.
-
No worries, I'm glad you were able to narrow down the cause this far already.
-
@stephenw10 Hi, Just to report back. Even after removing the Disk Widget the problem is still there cannot access the web gui at all then it starts working but can access different tabs. This is weird.
-
@stephenw10 Hi, Here is the Video, I had to put it on one of my Servers, Hopefully you will be able to play it. This is from one of my Firewalls in a different location, I have another with the same issue as well. (https://cloud.violetdragonscloudnetwork.co.uk/s/iCKFFgmqQ8jLQ5a)
-
What happens when you use the 'admin' user ?
Jack is fine, but that's probably not an 'admin'.
Dome info collected by the dashboard GUI page need 'admin' rights.Why creating a user like Jack ? pfSense is a router firewall, not some file server.
-
@gertjan Security, admin is a easy username to guess. Admin user does the exact same. But I have a ton of PHP-FPM processes when this happens. I've also triggered the problem by attempting brute-forces via SSH. I have va feeling that maybe something is going on.
-
@violetdragon said in pfsense 2.6.0 sshguard @ web gui bug/crash:
Security, admin is a easy username to guess.
Yeah, but who cares ?
Normally, the GUI should only be accessible from the LAN interface.
The other LAN ( OPT1 OPT2 OPT3 ) interfaces are meant to be used for you local network.
This allows you to even completely disconnect the LAN interface when you don't need the GUI access. That what security is.Easy user names and passwords are a thing on a public network.
Your LAN is not a public network.I've changed a user called '001' so it has admin privilges, like this :
Now I can login with '001' and the dashboard shows up in half a second.
What happens when you remove all or most of the the widgets ?
Login again to try again. -
Ah, OK so something you have on your dashboard is taking an age to timeout before even rendering.
We can't see what widgets you have showing there but can I assume it's the disks widget causing it? If you remove it then the dash displays in the expected time?
And does that only happen with user 'Jack'? If you login as admin do you still see the delay?Steve
-
@stephenw10 Hi, it happens on both users, admin and jack, Widgets i have on the home page I have also removed them then tested again but issue is still there, disabling ssh does not fix the problem, but there is loads of php-fpm in the logs.
Picture,
System Information,
Interfaces,
Services Status,
Installed Packages,
NTP Status,
pfBlockerNG,
Gateways,
UPS Status,
Firewall Logs,
Interface Statistics,
Thermal Sensors,
SMART Status,
Haproxy,
Snort Alerts,
Traffic Graphs.I have notice a something though, when this happens it's always from 12am to 5:30am when this happens. But i can trigger it though, false logins via SSH. But there's nothing in the logs about Brute Force attacks but there is something triggering it though because sshguard is being triggered and logs are full of sshguard every few minutes, disabling ssh in general does not fix the problem neither so it is either maybe a bug? or maybe someones inside the network.
Saying the time when this happens, I have noticed it's being triggered again but the Web Gui is still working, the logs are full of,
Apr 20 15:59:00 sshguard 84660 Now monitoring attacks. Apr 20 15:59:00 sshguard 11578 Exiting on signal. Apr 20 16:33:00 sshguard 85149 Now monitoring attacks. Apr 20 16:33:00 sshguard 84660 Exiting on signal. Apr 20 17:12:00 sshguard 57530 Now monitoring attacks. Apr 20 17:12:00 sshguard 85149 Exiting on signal. Apr 20 17:51:00 sshguard 26088 Now monitoring attacks. Apr 20 17:51:00 sshguard 57530 Exiting on signal. Apr 20 17:54:00 sshguard 34895 Now monitoring attacks. Apr 20 17:54:00 sshguard 26088 Exiting on signal. -
When you see this happen though the symptoms are that the dashboard takes a very long time to load but does eventually load. Other pages in the webgui load as expected?
Hitting the system with bad logins triggers sshguard but that prevents you accessing the firewall entirely for length of the block.
The sshguard entries you see in in the system log are the service restarting because your logs are rotating. Nothing is being blocked there.
This is unrelated to sshguard as far as I can see. It just produces similar symptoms.
Did you see which log is rotating at 3min intervals? What's in it?
Something is causing both a lot of logging and one of the dashboard widgets to load very slowly.
Some of those widgets include logs so that could be related. Or they could both be symptoms of something else.What made you think this was the disks widget at one time?
Steve
-
@stephenw10 Other tabs work except for the home page even removing the widget don't fix it, reason why i though it could of been the disk widget is that's when it stopped acting up when i removed it from the homepage,
I just find it weird how it happens only in the early hours of the night i noticed it because that's when i do my maintenance, upgrades etc on a weekend as it's not just me that uses the network.
I did some packet tracing not long ago but have not found anything unusual on the Network but i will run another tonight early hours of the morning and keep it running too see if i find anything obscure happening.
As for the logs rotating I just find it strange all the years I have been running pfSense I have not seen this activity until now which is why I think that there could be someone lurking in the network I might just be paranoid but it's always good to know what is going on with a network that is publicly exposed. As for the logs I have not checked which would you recommend to go with?
Not mentioned this but the zpool was not updated so i did that last night so ZFS is now running the latest update but I have noticed a ton of ZFS Write Errors in the System log but zpool status does not show any problems.
Thanks.
Jack.
-
You only see those sshguard logs since 2.6 because the logging was changed from circular logs to the standard FreeBSD logs. In previous versions the syslog service never restarted so sshguard did not.
-
@stephenw10 I see, could the Gui be kinda broken after upgrades ? Because this is a old install from 2.4 maybe that could be causing the problem ? It is strange though right?
-
I can't imagine anything that would have come over from the old config that would cause this.
The fact it only happens some of the time implies something external that's changing. It's going to be something that's trying to parse some huge file for some reason. Probably some log file given the sshguard logs.
I think you checked the Monitoring graphs for the incoming blocked traffic? That's what I expect to be the most likely cause.Steve
-
@violetdragon said in pfsense 2.6.0 sshguard @ web gui bug/crash:
Heres the output of,
( see above your /var/log/* files )
@violetdragon said in pfsense 2.6.0 sshguard @ web gui bug/crash:
21 -rw------- 1 root wheel 129542 Apr 14 16:07 system.log
41 -rw------- 1 root wheel 40644 Apr 14 01:49 system.log.0.bz2
33 -rw------- 1 root wheel 30645 Apr 11 20:35 system.log.1.bz2
33 -rw------- 1 root wheel 30312 Apr 9 07:00 system.log.2.bz2
41 -rw------- 1 root wheel 37309 Apr 6 22:58 system.log.3.bz2So, in less then one day, 15 hours, your system.log grows from nothing (it was rotated) to 130 Kbytes.
Mine is 10 Kbytes and it was created (rotated last February) !All you logs file are "big", you could set up, if you have the space for it, log files that are 10 times bigger;
There will be less rotation == syslog restating.
But keep in mind that the GUI - and other process, like sshguard, process (track) and read log files. Big files means more processing time.
PHP is nice for building web pages, but having it go through 'huge' files just to put a line with info on a screen wasn't its goal, and its not good at doing so. It becomes slow.Btw : this is me just thinking out loud.
I have just a small company behind my pfSense.
And a captive portal where hotel visitors use our Internet connection.Most of the sub folders like nginx, nut ntp are empty.
/pfblockerng/ has some files, nothing huge.Did you look into /snort/ ? This one, I don't like, as it can be set up to kill a system really fast, by 'eating' all disk and cpu resources.
No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
I had originally thought that sshguard reads the log files, and it can be setup to do that, but in pfSense it doesn't. Instead the syslog daemon pipes all authentication log entries to it directly.
But that still means sshguard is restarted whenever syslog does and that happens whenever any log rotates.Steve
