Can't access 3100 appliance
-
Yesterday I noticed some strange traffic on one of our servers. It was coming from an internal IP address. Upon checking further I found other strange things happening.
When I went to my pfSense 3100 appliance I found I couldn't log on. The page won't even load. I connected to the unit via the terminal cable. I had to reset the password, but then the page would load (surprised me since the page wouldn't even load before). A bit later however I couldn't access it again. Changing the password made no difference. I reset the WAN and then I could access the page.
As I was checking things out I discovered that I also can't access any package information. The packages appear to be there and working, but I can't access the Package information screen to check for updates etc.
This morning I came in and once again I can't log into pfSense again. I logged into the terminal and changed the password. No luck. I reset the WAN. No luck. I did a restore to a backup from before this mess started. I STILL can't log onto the GUI.
What's going on? And how do I fix it?
-
I did a reset to factory default followed by a restore of my last known good backup. I was still unable to log onto the GUI plus the unit was no longer passing any traffic to internal machines. I removed the appliance from my network and went to an older unit I'd kept as a spare so my new appliance is no longer in use until I find a way to get it working again.
Interestingly enough, my old unit won't let me log in either. The page comes up but it says my password is invalid. I double checked the password (I have a program that stores them) and it's correct. I'm hesitant to try to reset it at this time since the 3100 is already down and I don't have a second backup.
-
I did another factory reset. I went through the setup process and it was up and connected to the GUI. I did a restore from a backup file from December which had been stored on a different system where it was untouched. The restore was successful. I was able to log into the GUI at the correct WAN address and move around in the GUI as normal. And then after several seconds, it went dead and I could not connect again. Nothing I do at that point allows me to get into the GUI again. I'm lost.
-
It sounds like you have some sort of address or subnet conflict.
Try running option 13 at the console menu to check for updates and let is see whatever error is shown there. If it's unable to check for packages it probably can't reach the update server. It's probably some network issue since it's also preventing you access the gui.
Steve
-
@stephenw10 It checks for updates fine from the console. More confusing... I got the unit up and running correctly and noted the steps I took to get there (factory reset, connect lan to computer, run setup, when it reboots move lan to network switch). At that point I could log into the unit via the WAN IP and everything was working great.
So I went to change the password since I don't know how the hacker got in yesterday to take down pfSense in the first place. I changed the password in the GUI and hit save. I then logged out and tried to log back in again however the page won't load at all. I can't get the log-in screen. Tried multiple machines even.
So.. I started over. Went to factory settings and repeated the exact same process. However.. I still can't get to the GUI. It worked the last time, but not this time even though I followed the exact same procedure. It's passing traffic to the switch fine. I just can't get into the GUI.
-
From your description I think it's very unlikely you were 'hacked'. It sounds like you just lost access to the webgui.
Were you able to access it via SSH or ping it? Are you able to do that now you have lost access again?
It seems far more likely you have some LAN side conflict. Try disconnecting the LAN again and then connecting to the WAN IP externally. Or disconnecting the WAN and connecting only from the LAN.
Steve
-
@stephenw10 The trouble is there were entries on our FTP site from internal IPs from things such as our shipping computer. And someone shut down several of our VMs and even changed the password on our host system so we couldn't log in without changing it at the root. Were it not for those things I might agree with you.
Currently I can ping the 3100 unit. I can get it to open in Putty however when I try to log in it just says Access denied. I can't connect to the GUI but I still have my notebook connected via the serial port.
I can't connect via the LAN because I'm in Pass-through mode.
If I bypass pfSense and go to the WAN I get the log-in screen however once I enter my pass word it just sits there "waiting for ......" and then after bit it says :
CSRF check failed. Your form session may have expired, or you may not have cookies enabled.
Try Again
Debug: sid:6f7a5d40a0c7103fa2bc2e9f9ac212a5f2f819b7,1578508171I've tried from multiple computers with two different browsers. Same error.
-
@stephenw10 And boom! Out of nowhere the page just loaded. I'm afraid to do anything. I looked at the logs and realized the times were all way off so I went to settings and changed the NTP and clicked Save. It's been spun for several min. afterward before finally saying the changes had been applied. I'm still waiting for the Dashboard to reload.
-
@stephenw10 And it's gone again.
-
Hi @cdsJerry , Internet is functioning otherwise though? On a mailing list I'm on someone posted these two messages several weeks ago:
"Anyone experiencing issues logging into a pfsense router this morning. We had 18 routers start this last night. The only thing that stops working is logging into the console, and openvpn users logging in with local or LDAP accounts. Everything else is working including site to site vpns. After a reboot you can log in for about 15 mins and then the problem returns. We are working with Netgate and are trying something they suggested...."
and
"We found the fix. The router has to be rebooted and you have 15 min to login before the console locks up again, enable SSH and then connect with Putty and run this command: pkg upgrade -f pfSense-repo"We didn't experience this problem on any pfSense I've logged into, in the past few weeks, so I'm not sure what was different about their setup. But if it sounds the same as your issue this may be a fix.
-
@teamits Yes, the Internet is working fine otherwise. The only symptom I have now is that I can't connect to the GUI. I did a reboot a couple of time but can't log in even after the update. I tried to do the update from console (which worked earlier) but now that's not working either.
I tried your suggestion. After entering the command it responds with "Updating pfsense-core repository catalog... and then just sits there. I had to throw and interrupt at it to get a prompt back (and then I rebooted again).
However there must be some truth to it. Once it rebooted I've been able to get into the GUI normally. Everything looks good, at least for now. The only thing is, I'm still connecting to the GUI from OUTSIDE of the pfSense protection network. Just as soon as I move inside the pfSense protection... I lose access to the GUI again. It's as if pfSense is blocking its own traffic.
-
@cdsJerry said in Can't access 3100 appliance - hacked:
It's as if pfSense is blocking its own traffic.
It can...it will do what it's told. Check System/Advanced, that "Disable webConfigurator anti-lockout rule" is not checked, and/or add firewall rules allowing traffic to pfsense:443 on that interfce.
-
@teamits The check box isn't checked. And scrolling down a bit further I see my network IP is also listed on the Whitelist under "Login Protection". The radio button at the top is set to https. Is the system not smart enough to open 443 when that's selected? It shouldn't really matter because 443 is open for access to the servers anyway.
-
@cdsJerry ... and lost access again. <sigh> This thing was running for a couple of months with no problem and now it won't run for more than a few min. It's frustrating, especially since we know it didn't stop someone yesterday. Without the GUI up we don't even have a way to monitor it.
-
@cdsJerry The anti lockout rule does add firewall rules on LAN to allow access. Disabling it (checking the box) removes them.
From SSH can you restart webConfigurator and/or PHP and gain access? -
The problem description sounds either like it's churning on something badly using all the available cpu cycles. Perhaps a flapping link or something stuck in a loop. Try running
top -aSHat the console.Or it's a bad route or subnet conflict or something similar.
Check the routing tablenetstat -rn4.
Check the system logs for reported IP conflicts or any other errorsclog /var/log/system.log.Steve
-
@stephenw10 This morning I can't log into the GUI even from outside. And when I try to connect via SSH it tells me my password is invalid. I did a copy/paste from my password manager software. I'm using the correct password.
top -aSH from the console shows the unit idle 98% of the time. The other items that show up all look correct. I watched it for a while and nothing seemed out of place.
netstat -rn4 showed
default the IP as my gateway IP
The DNS IP
The Network ip/28
The pfSense IP
The local IP
Nothing more (I'm in pass-through mode)The clog command set off a long list and I didn't know how to stop the scroll but when it finished I rolled back up the list. There were a ton of entries telling me I need to read the license file.
There was also a line 12cache0: cannot allocate IRQ not using interrupt
And another line etc/rc.d/hostid Warning unable to figure out a uuid from DMI data, generating new one -
@cdsJerry said in Can't access 3100 appliance - hacked:
This morning I can't log into the GUI even from outside. And when I try to connect via SSH it tells me my password is invalid. I did a copy/paste from my password manager software. I'm using the correct password.
Do you mean by "... can't log into the GUI even from outside." that you have the firewall GUI available on the WAN port? If you do, and you don't have a VPN in place, then almost certainly your system could have been severely compromised.
Take the firewall out of service for a moment, restore to factory defaults and then import your last known good backup. Do all this initially connected with just the serial console cable.
Next, plug a laptop or a local PC directly into the LAN port, give the PC an IP address within the LAN port subnet (if necessary), and see if you can login. Things may be very slow because the GUI will be trying to contact the pfSense update servers to check for the latest firmware. Even still, this will let you verify your passwords are good.
Once in, then connect the WAN connection and see how things go from there.
Your description really sounds like a badly hosed configuration. And are you sure that someone has not put another device on the network that has the IP address of the firewall? That could cause the issues you are seeing.
-
@bmeeks said in Can't access 3100 appliance - hacked:
Do all this initially connected with just the serial console cable.
My pfSense is configured in pass-through mode. The GUI has to sit on the public IP as there is no LAN IP in pass through mode. The configuration was done with Netgate support so I'm pretty confident it was configured correctly and that's the backup I've been restoring to. There are multiple WAN IPs that pass through the system on their way to various servers. There is a second firewall between the public IPs and our internal network so this first pfSense unit is just limiting and cleaning traffic that's headed for the servers. Again, I'm pretty confident in it's configuration. I can't explain it's recent behavior unless and update changed something.
-
@cdsJerry Do you have antivirus in place to prevent keyloggers and such being installed on your public servers? Connections to the FTP from an inside computer normally not expected I would also find concerning. At one time we found a repository of German DVD rips on our main (inside) file server because of a lousy password policy on some test accounts (not admin/admin, but just as bad) accessed through our Citrix web access. Maybe someone got a phishing email or brought in a compromised flash drive. In my experience I've found the simply baffling problems generally caused by malware. I might first check with your users who have the auth to manage your virtual environment. Sounds like an admin's creds got out.
