Can't access 3100 appliance
-
@cdsJerry The anti lockout rule does add firewall rules on LAN to allow access. Disabling it (checking the box) removes them.
From SSH can you restart webConfigurator and/or PHP and gain access? -
The problem description sounds either like it's churning on something badly using all the available cpu cycles. Perhaps a flapping link or something stuck in a loop. Try running
top -aSH
at the console.Or it's a bad route or subnet conflict or something similar.
Check the routing tablenetstat -rn4
.
Check the system logs for reported IP conflicts or any other errorsclog /var/log/system.log
.Steve
-
@stephenw10 This morning I can't log into the GUI even from outside. And when I try to connect via SSH it tells me my password is invalid. I did a copy/paste from my password manager software. I'm using the correct password.
top -aSH from the console shows the unit idle 98% of the time. The other items that show up all look correct. I watched it for a while and nothing seemed out of place.
netstat -rn4 showed
default the IP as my gateway IP
The DNS IP
The Network ip/28
The pfSense IP
The local IP
Nothing more (I'm in pass-through mode)The clog command set off a long list and I didn't know how to stop the scroll but when it finished I rolled back up the list. There were a ton of entries telling me I need to read the license file.
There was also a line 12cache0: cannot allocate IRQ not using interrupt
And another line etc/rc.d/hostid Warning unable to figure out a uuid from DMI data, generating new one -
@cdsJerry said in Can't access 3100 appliance - hacked:
This morning I can't log into the GUI even from outside. And when I try to connect via SSH it tells me my password is invalid. I did a copy/paste from my password manager software. I'm using the correct password.
Do you mean by "... can't log into the GUI even from outside." that you have the firewall GUI available on the WAN port? If you do, and you don't have a VPN in place, then almost certainly your system could have been severely compromised.
Take the firewall out of service for a moment, restore to factory defaults and then import your last known good backup. Do all this initially connected with just the serial console cable.
Next, plug a laptop or a local PC directly into the LAN port, give the PC an IP address within the LAN port subnet (if necessary), and see if you can login. Things may be very slow because the GUI will be trying to contact the pfSense update servers to check for the latest firmware. Even still, this will let you verify your passwords are good.
Once in, then connect the WAN connection and see how things go from there.
Your description really sounds like a badly hosed configuration. And are you sure that someone has not put another device on the network that has the IP address of the firewall? That could cause the issues you are seeing.
-
@bmeeks said in Can't access 3100 appliance - hacked:
Do all this initially connected with just the serial console cable.
My pfSense is configured in pass-through mode. The GUI has to sit on the public IP as there is no LAN IP in pass through mode. The configuration was done with Netgate support so I'm pretty confident it was configured correctly and that's the backup I've been restoring to. There are multiple WAN IPs that pass through the system on their way to various servers. There is a second firewall between the public IPs and our internal network so this first pfSense unit is just limiting and cleaning traffic that's headed for the servers. Again, I'm pretty confident in it's configuration. I can't explain it's recent behavior unless and update changed something.
-
@cdsJerry Do you have antivirus in place to prevent keyloggers and such being installed on your public servers? Connections to the FTP from an inside computer normally not expected I would also find concerning. At one time we found a repository of German DVD rips on our main (inside) file server because of a lousy password policy on some test accounts (not admin/admin, but just as bad) accessed through our Citrix web access. Maybe someone got a phishing email or brought in a compromised flash drive. In my experience I've found the simply baffling problems generally caused by malware. I might first check with your users who have the auth to manage your virtual environment. Sounds like an admin's creds got out.
-
@provels We do run antivirus as well as malware and other detection software. I re-ran everything manually and didn't find anything.
Indeed some of the FTP accounts use some pretty lousy passwords but at the same time they're limited to single directories which are usually emptied as we complete jobs. I checked every directory on the server (that took a while) and they are all empty. Actually I went a step further and shut down the FTP server. We find it's not used much any more since there are so many file-share services from Google, Dropbox, etc. and people understand them better than FTP. I think it's been months since someone actually used FTP to send us files, so I shut it down.
I'm always amazed that people have FTP servers out there where a folder had both read and write permissions with public access. They're just waiting to become porn storage.
-
@cdsJerry said in Can't access 3100 appliance - hacked:
@bmeeks said in Can't access 3100 appliance - hacked:
Do all this initially connected with just the serial console cable.
My pfSense is configured in pass-through mode. The GUI has to sit on the public IP as there is no LAN IP in pass through mode. The configuration was done with Netgate support so I'm pretty confident it was configured correctly and that's the backup I've been restoring to.
It might be time to pay Netgate Support again and open a case directly with them. Might take some effort to provide them some method of remote access to the firewall appliance itself. Maybe use one of the available Windows remote desktop web platforms to give Netgate Support access to a Windows PC where the SG-3100 is connected to it locally via the console cable.
-
@bmeeks I know how to give them direct access, well, when anyone has access, which seems to be for only a while after a reboot. It's all about the money at this point as our business is selling CDs, DVDs, SD cards, and USB drives. The CD/DVD side just isn't the hot ticket it once was so the purse strings are incredibly tight.
-
if you have the WAN and LAN bridged and only a public IP on the firewall you will, as you say, have to use that IP to access it but that doesn't mean it needs to be open to the internet. You can still set firewall rules to restrict access to the webgui or ssh to internal clients or known external IPs only.
Steve
-
@stephenw10 I agree with the firewall to restrict access to the GUI, and I think my rules are already set for exactly that. But my current problem is that I can't access the GUI myself so I can't get in to even verify that. And if it wasn't set, why would I have access for 15 or so min. after the reboot? I didn't have this issue before the reload so the settings shouldn't have changed.
-
15mins is suspiciously like an ARP timeout. Maybe you have some IP conflict and something else is responding to ARP when it times out breaking your connection.
If you are inadvertently connecting to something else that might explain why your password seems to stop working.
The SSH key would be different though, the SSH client should warn you about that when you try to connect.If you restart php and the webgui at the console (menu options 16+11) do you get connectivity back?
Steve
-
@stephenw10 I haven't connected since Friday but when I went to it today, it connected just fine. I've been logged on for over 30 min. now. I can move around all the menus and options normally. It loads fast. I can make changes.
However, if I make a change and hit save it stalls out saying it's sending request. But if I then click off to another menu and come back, the change has been saved. And I can't access the Dashboard at all. It just says Waiting for xxx.xxx.xxx.xxx... but never loads. Not sure if this is related to the same issue or if this should be a new thread.
-
@stephenw10 I left it trying to load that page and came back later. The page had loaded at some point so I went off of that page to another page then back to the Dashboard. It's still trying to load the Dashboard again.
I also see it loaded a new version about 45 min. ago.
-
That sounds more like a general connectivity issue. When you go to the dashboard depending on which widgets you have there, it reaches out to check several things on the internet, the firmware update check for example.
If it cannot connect to those they have to timeout. The dash will be much slower to load in that situation than other pages.
If you have ACB configured then it tries to save a backup everytime you make any chnage to the firewall and that can be a problem if there is no connectivity.
Make sure all your configured DNS servers are responding in Diag > DNS Lookup.
Make sure it can ping out in Diag > Ping.Steve
-
@stephenw10 The page loaded instantly this morning. I went to the Diag > DNS Lookup and did some lookups. They came back with 6-11ms return times. I'm on fiber and haven't seen any connectivity issues on any of the servers.
Yesterday when I'd try to load the Dashboard and it would eventually complete, it showed the "Netgate Services and Support" as just a spinning star as if it was trying to get an update. Today it loads with the rest of the page so that's different. I didn't make any changes however.
Today it appears to be running fine.
-
Mmm, some IP conflict would tie in with that if whatever device it was has now been removed or given a new IP.
I would expect to see something logged in pfSense reporting another device using the same IP though.
Steve
-
@stephenw10 And these are all fixed IPs and no changes to any of them during this time.
-
I think you should edit the subject of your first post to remove the word "hacked".
-
@cdsJerry said in Can't access 3100 appliance - hacked:
@stephenw10 And these are all fixed IPs and no changes to any of them during this time.
Are you 100% sure, though, that another transient device is not getting connected to the network and then later removed, and this transient device happens to have the same IP address as your pfSense box? The symptoms you describe have the hallmarks of a duplicate IP address on the local network. This transient device could be a wireless device or a wired device (someone's laptop maybe).
-
@NollipfSense Thanks for the suggestion. That is a "hot" word. It does appear that someone did get into our network as they connected to the computer used to do our shipping and they shut down several VM machines. It looks like they booted themselves out when they shut down the machine they were using to access our network. The point is.. they got past the firewall somehow and when we went to look at the firewall we found the password appears to have been changed so we think we were indeed "hacked".
-
@bmeeks said in Can't access 3100 appliance - hacked:
@cdsJerry said in Can't access 3100 appliance - hacked:
@stephenw10 And these are all fixed IPs and no changes to any of them during this time.
Are you 100% sure, though, that another transient device is not getting connected to the network and then later removed, and this transient device happens to have the same IP address as your pfSense box? The symptoms you describe have the hallmarks of a duplicate IP address on the local network. This transient device could be a wireless device or a wired device (someone's laptop maybe).
There is no wireless access to any fixed IPs or device that has dhcp to any fixed IPs. The only way to access those IPs would be to connect to the managed switch, or the pfSense appliance itself. In our tiny company I'm the only one with access to those pieces of equipment. I didn't make any changes except I swapped out our old pfSense for the appliance while I rebuilt the appliance. The old pfsense and the appliance are not on the same WAN IP so there shouldn't have been a conflict there. And both devices were never both connected at the same time for that matter.
It's still working OK. I've had it open all day today and no glitches in spite of no changes.
-
@cdsJerry What you described seems more internal...like a disgruntled employee who knew the network administrator's password and paid back...shame on the network administrator indeed!
-
@NollipfSense There are only two of us and I'm the only one with access. I use secure passwords and have never shared those passwords with anyone. I use a password manager (Dashlane) to keep track of them because I use comlex passwords that are never used in more than one place. My one employee has zero access to pfsense.
-
@cdsJerry At least, you know it was the shipping computer that was used; however, it still puzzling because a complex password is not easy to change on a firewall, much lest a robust firewall such as pfSense. So, do you know what IP address was used, the time and date and the ISP the IP address came from? Is your password manager configured to change the password after a period elapsed? Do you have any idea why you were targeted?
I have never used a password manager on a firewall. I still think you should remove "hacked" until you're absolutely sure with a preponderance of substantiable evidence.
-
@NollipfSense I do not know what IP address was used. The shipping computer is connected to the LAN and has no WAN IP. It's behind pfsense and behind another router. I can only guess at the time based on when I noticed an attempt to log into our FTP server (from the shipping computer). They may have been inside for a while before that of course.
The password manager doesn't change the password on pfsense, nor is it connected to it. Dashlane is simply an encrypted password management program that creates and stores secure passwords. Google it, it's really handy. To change the PW on pfsense I'd still need to log into it via the GUI. Dashlane just allows me to use longer more secure passwords without trying to remember them all.
I have no idea why I'd be targeted. Our domain name gets a lot of hits but we're a small company. There are no financial fortunes here to discover. But a hacker wouldn't know that until he gets in.
And I did remove "hacked" from the subject already based on your first suggestion.
-
@cdsJerry said in Can't access 3100 appliance:
And I did remove "hacked" from the subject already based on your first suggestion.
Cool...I didn't notice as I was at the bottom of the thread...thank you!
I got to say though the shipping computer with no WAN IP made me scratch my head...so, what the router in front of it but behind the pfSense box do?
-
@NollipfSense My pfsense is in pass through mode. It doesn't issue IPS etc. It just makes sure the traffic coming in is "clean" and controls what ports are open etc. The WAN IPs all pass through it to their destinations which then control the traffic from that point forward.
-
Given that someone was able to shutdown VMs and that your firewall is in pass thoughmode with public IPs internally, is it possible 'they' created a new VM on the same IP as pfSense? Or altered the IP of existing VM?
Steve
-
@stephenw10 There weren't any new VMs. We only have a few running so it would have been easy to spot. And the VMs aren't on the same WAN IP as pfsense. They pass through pfsense but you can't access pfsense from them as they are not the WAN assigned to pfsense. Pfsense is the only thing using that IP. The VM host uses a LAN IP for it's access. It is connected to the WAN in order to pass those WAN addresses over to the virtual NIC cards in the VMs but none of those WAN IPs are assigned to the NIC on the host (Proxmox). None of the IPs on the VMs have been altered.
-
@cdsJerry said in Can't access 3100 appliance:
@NollipfSense My pfsense is in pass through mode. It doesn't issue IPS etc. It just makes sure the traffic coming in is "clean" and controls what ports are open etc. The WAN IPs all pass through it to their destinations which then control the traffic from that point forward.
Okay, I remember reading your response earlier where you mentioned the above. I don't know what to say...
-
I'm assuming you've gone back through the pfSense logs and there are no reports of: 'xxxxxx is using my IP!' ?
Because it really looks like that might have happened from everything you describe.
Steve
-
@stephenw10 The short answer would be no we didn't. I didn't see anything that jumped out at me when I was looking in the logs but I didn't know what I was looking for. In pass-through mode would that error even show up?
When we couldn't get the password to reset and we couldn't get into pfSense via the GUI we ended up re-loading the entire thing from a backup just to make sure that none of the rules or aliases had been altered, so everything was reset at that point to try and secure the network again.
-
pfSense will log that if it sees some other device using it's IP so broadcast messages from that IP or something else responding to ARP. I would see that traffic in a bridged setup still.
Steve
-
@stephenw10 I don't see anything like that in the logs. And it's still working fine today. Nothing on the network has changed but it's all operating as expected for several days now.
-
Hmm, well that's a public IP so it could have been some issue at your provider. They mistakenly issued your IP to another client perhaps and have now corrected that. Hard to say at this point.
-
It definitely looks like your configuration is more complex, so maybe this is not very useful to you.
But I'll toss it out there.One of the practices I've developed is to assign virtual IPs to my router.
I worked a fair bit in telecom before as a developer, and I just developed this as habit in our test labs.
I've kept it going in my home setups.
For example, my sg-3100 has 192.168.1.1, 192.168.1.2...The reason I do this is in case a device I plug in to the network default to a certain conflicting IP. This way, I can still access the router and see what's happening.
-
@yaminb In my case there's no potential for an IP conflict because there's no DHCP on any WAN IP. The pfSense has a WAN but everything else is just passed along. The routers are all downstream and would hand out DHCP to any device plugged into their networks. I only have a dozen WAN IPs so it's not hard to track those in the switch, and nothing would ever be connected directly to the ISP other than those dozen, and even those are post-pfSense.