Diagnostic - Cleaning Up After Being Hacked
-
Hello!
I have been using PFSense for years and recently started hosting a web server. I never once had any issues with malicious activity on my network until I started hosting the web server. I just received notice from my ISP that their has been malicious activity originating from my IP address. They suspected a brute force attack via an open SSH port.
Looking over my configuration, the only place I had an open SSH port was on my web server. I'm assuming that is where the attack originated from. In hopes to isolate the issue, I was hoping to get into my router and see if I could find where the originating IP (not that it would help) was AND to see how to better fortify things and move on. Needless to say, my ISP turned off my connection until I addressed the issue. Also my ISP gave me zero information about the issue. Other than it appeared to be from an open SSH port AND that the suspecting hacker was sending thousands of packets a second out from my connection...
I cannot seem to log into the Webconfigurator. But I did find I could get this resolved by resetting the password on the console and resetting the web configurator. I will try that when I get home.
I have also downloaded the latest stable version of PFSense just in case I have to format and start over. I did have a bunch of preventative addons on my firewall that certainly helped for ad blocking and filtering from that perspective.
So aside from the obvious.... I plan on changing all passwords, closing all open ports, locking down root access to firewall internally and externally, and putting my web server in the DMZ. Are their any utilities within PFSense that would flag suspicious activity before an issue like this happened?
Thanks!
Cheers,
Mountbaldy
-
Did this happen on your server or pfSense? If you don't know, you can't fix it. If either has been compromised, then reformat and reinstall. That's the only way to be sure you've cleared it.
-
@Mountbaldy said in Diagnostic - Cleaning Up After Being Hacked:
Hello!
I have been using PFSense for years and recently started hosting a web server. I never once had any issues with malicious activity on my network until I started hosting the web server. I just received notice from my ISP that their has been malicious activity originating from my IP address. They suspected a brute force attack via an open SSH port.
Looking over my configuration, the only place I had an open SSH port was on my web server. I'm assuming that is where the attack originated from. In hopes to isolate the issue, I was hoping to get into my router and see if I could find where the originating IP (not that it would help) was AND to see how to better fortify things and move on. Needless to say, my ISP turned off my connection until I addressed the issue. Also my ISP gave me zero information about the issue. Other than it appeared to be from an open SSH port AND that the suspecting hacker was sending thousands of packets a second out from my connection...
I cannot seem to log into the Webconfigurator. But I did find I could get this resolved by resetting the password on the console and resetting the web configurator. I will try that when I get home.
I have also downloaded the latest stable version of PFSense just in case I have to format and start over. I did have a bunch of preventative addons on my firewall that certainly helped for ad blocking and filtering from that perspective.
So aside from the obvious.... I plan on changing all passwords, closing all open ports, locking down root access to firewall internally and externally, and putting my web server in the DMZ. Are their any utilities within PFSense that would flag suspicious activity before an issue like this happened?
Thanks!
Cheers,
Mountbaldy
Close the NAT to the server while fixing it.
Format the server, then reinstall it and then open NAT again.
What is running on the webserver?
-
@JKnott
It appears that the only thing compromised was the web server. I haven'd dug into the web server yet. I just unplugged it and will put it on the bench here shortly.@Cool_Corona
I reset PFSense to factory defaults. Unplugged the web server for now and cleared all NAT entries I had (which was only the web server and ssh on web server). I did see some entries on my firewall that were flagged by PFSense as malicious intent from someone trying to access my admin console on the PFSense box. I appears PFSense locked them out after too many unsuccessful attempts. I don't know if they ever did get into my PFSense box as the only thing I was locked out of was the webconfigurator and I was able to regain access by resetting my admin password on there.I'm running CentoOS on the web server running Apache and Joomla.
Thanks for the input!!
Cheers,
Mountbaldy
-
Hi,
I understand your concern, if they hack you up, it always sucks....
(it can mean a lot: you are a real victim or you weren't careful enough)- we have been dealing with web hosting for a long time
I know everyone has different ideas, but let me offer a proven configuration for the web server operation (it only requires a penny investment from the OSs / install parts)
first step & suggestion:
- never leave SSH port(s) open this is a rule of thumb or hide it very well (over 40 - 50K) or follow what I have described below:
• OS: Debian 10.x (Buster) 64bit
• Apache Worker, factory package
• for WAF - Mod Security apache module with OWASP rules, factory package
• PHP-FPM 7.3 or rather 7.4 if it goes with everything but definitely 1 version
• PHP can only write where we allow it, ie it stays on the www-data user
• firewall inbound to CF IPs is limited to http and https, just as SSH access is also severely limited (http can be completely
disabled by likely, CF solves http-> https redirect)
• SSH access is password protected only (or with Cert.) ++++edit: I mean just DO NOT leave it open!!!
• firewall to the outside, by default everything that is needed (external APIs and their counterparts) is enabled separately
• hosting-type access via SFTP, SSH, although shell access may be possibleCF = CloudFlare (https://www.cloudflare.com/plans/)
put the web server in DMZ and protect it as much as possible, such as WAF
pfSenses can't do much in front of a real web server, there is no absolute security
and pray .... this is true, bad guys are looking for weak pont on the internet.....always
PS:
most importantly, find out exactly what they broke - pfSense or web server? - we have been dealing with web hosting for a long time
-
@DaddyGo said in Diagnostic - Cleaning Up After Being Hacked:
SSH access is password protected only (or with Cert.)
I thought the secure way was to use a public/private key pair.
-
Thank you for such a detailed report. I appreciate all of your advice! I'm new to the hosting game but am learning very quickly.
Since I disabled my webserver, I have not noticed any odd traffic on my network. I also installed Snort to try and keep a better handle on things. I will be purchasing an additional NIC to get my webserver onto a dedicated DMZ. I will harden it off from there.
I do believe that the webserver got compromised. I could see traffic originating from an odd local IP address trying to gain access to my PFSense box. I believe they never did gain access and PFSense turned off the Webconfigurator.
You prefer Debian over CentOS?
Thanks again! I'm learning by trial by fire. Thankfully my webserver is for my side gig (non technology related) and I am not pressed for getting that back online right away. But... I need to dig more into it so I can figure out what was done.
Cheers,
Mountbaldy
-
@JKnott
yes, the "only" here is a bit ambiguous+++++if your settings are good, you can quickly ban unsolicited accesses, like SSH scanners
-
Hi,
it's nothing, but really...we help where the experience can be important and also to share
I would do two more important things:
I suspect they have achieved more than you might think...
(just imagine, going into a supermarket, but just looking at the mineral water class part, while there are many other attractive products in the store)- ask your ISP not to share this compromised IP with you or anyone else for a while.
-if it was a botnet or similar attempt, then it is a known IP in front of the badguys
-don't start web server until ................ you know
Yes, absolutely Debian, on this theme!
- ask your ISP not to share this compromised IP with you or anyone else for a while.
-
Hello!
You may want to explore using a simple webops package, like plesk, on your server behind pfsense. It can help immensely with web server/security management/maintenance.
John
-
When you have any public-facing server, keeping all the software elements installed on that server patched and updated with the latest security hotfixes is paramount!
If possible, put your web server software/site into a chroot jail so that even if a hacker breaks out to the shell of the web server, he is still limited by the chroot environment.
Make sure that all remote access is via secure encrypted means only. That means SSH must be configured to ONLY USE a public/private key pair. No password access! No matter how "complex" and "secure" you think your password is, it is never as good as a public/private key pair.
Make sure that remote admin access to public-facing servers is limited in some fashion to specific IP addresses. The best way to accomplish this is via a VPN. The assumption here is you as the admin need remote access from outside the LAN where the server resides. If you can do your server administration tasks locally, then ditch all remote access from the Internet and make it local LAN only. Still use SSH only and with a public/private key pair, though!
Use firewall rules to control traffic to and from the web server. For example, if the web server needs DNS, limit it to accessing only your local firewall DNS server and allow it to go nowhere else for DNS. That would limit a favorite "hacker control channel" exploit that attempts to disguise such traffic as normal port 53 DNS lookups. Limit those DNS lookups to only a server you control to deny that avenue.
Finally, putting an IDS such as Snort or Suricata on the DMZ interface where the web server is attached can help identify certain attacks. Although, with more and more of web traffic these days being SSL (https), the ability of the IDS to see into the payloads (so called deep packet inspection) is becoming more and more limited due to the encryption.
-
Also :
A web server, the actual process, should be robust.
All latest versions of Apache2, nginx etc should be able to dodge requests that "make no sense".
You can see them : have a look at the web server's request logs and error logs. These are often used by tools like fail2ban that recognized known attack, and block thes IP for the future.When you run an off the shelves CMS liike Joomla, Wordpress, or e107, or PHPBB, etc, this can be detected by the "visitor". He will then use special known requests that would help him to gain access to your web server.
It's a rather classic type of attack, and would work if you used ancient version of the web server, it's dependencies or its the CMS.
Be careful : the default CMS is often pretty rock solid, but .... the so called plugins / add ons /extension are NOT. They are written by people like you and me, who don't no sh*i about SQL injection, etc. SO the angle of attack is often focussed on this addon / extension / ....
Only install and use an addon if your are sure it doesn't contain any flaws. If you are not sure, just don't install it.As a web server admin, you should be able to explain every line in the web server log file. For every line you should know if something has to been done, or not. A web server isn't a thing that you activate, and leave up running. It's an never ending job.
Btw : I'm using myself the classic Debian OS, the classic Apache2 and fail2ban as the only protection. There is no firewall on my dedicated server. This server is now up for more then 20 years (I migrated to 'bigger' ones a couple of times).
Because the incoming traffic is https on port 443 (port 80 is about 2 % of the traffic these days) and the firewall can't inspect SSL traffic anyway. I use mostly bare bone CMS's and keep the number of addons limited.
I never even bothered to use tools like snort / suricate /squid / etc which need a 24/24H surveillance ... -
Thanks again everyone for your updates to this thread.
It looks like I have a bit of homework to do to get things caught back up on my web server, but I knew this. My web server and firewall were both up to current on OS patches. I will be doing more digging on the webserver side of things this week as time allows. For now it's shut off and PFSense is doing its job.
I do probably pay more close attention to my firewall than the average person. I try to go in and update it periodically to the most current version. I also do look at the logs on there and a do my best to make sense of any entries that don't look like normal traffic. When things were not normal, I was busy and let them be. I should have addressed the issue as soon as I noticed something. PFSense was telling me things weren't OK. I just ignored them.
I did add Snort to my firewall for some added monitoring. So far it hasn't picked up anything unusual.
@Gertjan
I do run Joomla with an addon. I'm running shaper_helixultimate. I run their out of the box version. Hopefully, shaper_helixultimate isn't a bad addon.I believe I will be installing fail2ban once I get this thing resolved. It sounds like that is a good option to help prevent this kind of thing.
Cheers!
Mountbaldy