PFSense blocking SSH access

jolebole · Sep 5, 2015, 7:15 PM

Hello all. I have a pfSense running for the past year and its been great. I recently wanted to access it through SSH and i read that I need to enable it from the advanced page, which I did, but I cant login at all. I tried from my win and my mac computers and the connection drops instantly. It doesn't even prompt me for a password. What can be the cause of the problem? The ssh service is runing and I tried both users admin and root. pfSense ver.2.2.2

Thanks.

NOYB · Sep 5, 2015, 7:45 PM

Are you accessing via LAN or WAN? Check for firewall rules that may be blocking. Also have you enabled anti lockout. That will create a pass rule for port 22 at top of LAN firewall.

jolebole · Sep 5, 2015, 8:57 PM

I am accessing from the LAN. Anti-lockout already has a top rule to enable port 22. SSH Lockout table is empty

johnpoz · Sep 5, 2015, 11:00 PM

"It doesn't even prompt me for a password."

So you get prompted for username, and then it disconnects sounds like you have password auth turned off - which is good but you would have to setup public key auth. Did you setup public key auth??

jolebole · Sep 6, 2015, 2:05 AM

Well from my Mac I do " ssh root@10.0.1.1" and I dont get a prompt for a password, the connection closes instantly. From putty in Windows I get the login prompt, but as soon as I enter root and hit enter, the connection drops. The Disable password login for Secure Shell option is disabled. I never had to login with ssh before, so I have not changed any options regarding ssh or any firewall rules. I was never able to login.

johnpoz · Sep 6, 2015, 2:47 AM

disabled so you have a check box or don't have a check.. If you have a CHECK then password is disabled an you have to use public key.

jolebole · Sep 6, 2015, 5:18 AM

I do not have a check box. Here is the output

macbook-air:~ user$ ssh root@10.0.1.1
The authenticity of host '10.0.1.1 (10.0.1.1)' can't be established.
RSA key fingerprint is 41:3e:39:85:59:2b:4c:2f:b8:08:f8:e6:0b:df:4b:8f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.1.1' (RSA) to the list of known hosts.
Connection closed by 10.0.1.1
macbook-air:~ user$

Derelict · Sep 6, 2015, 6:56 AM

What's in Status > System Logs > System Tab when you try to connect?

jolebole · Sep 6, 2015, 7:00 AM

This is the system log:

Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: in openpam_check_desc_owner_perms(): /etc/pam.d/login: insecure ownership or permissions
Sep 6 01:59:03 login: in openpam_check_desc_owner_perms(): /etc/pam.d/login: insecure ownership or permissions
Sep 6 01:59:03 login: pam_start(): system error
Sep 6 01:59:03 login: pam_start(): system error
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: in openpam_check_desc_owner_perms(): /etc/pam.d/login: insecure ownership or permissions
Sep 6 01:59:03 login: in openpam_check_desc_owner_perms(): /etc/pam.d/login: insecure ownership or permissions
Sep 6 01:59:03 login: pam_start(): system error
Sep 6 01:59:03 login: pam_start(): system error
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: in openpam_check_desc_owner_perms(): /etc/pam.d/login: insecure ownership or permissions
Sep 6 01:59:03 login: in openpam_check_desc_owner_perms(): /etc/pam.d/login: insecure ownership or permissions
Sep 6 01:59:03 login: pam_start(): system error
Sep 6 01:59:03 login: pam_start(): system error
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 login: in openpam_check_desc_owner_perms(): /etc/pam.d/login: insecure ownership or permissions
Sep 6 01:59:03 login: in openpam_check_desc_owner_perms(): /etc/pam.d/login: insecure ownership or permissions
Sep 6 01:59:03 login: pam_start(): system error
Sep 6 01:59:03 login: pam_start(): system error
Sep 6 01:59:03 init: getty repeating too quickly on port /dev/ttyv0, sleeping 30 secs
Sep 6 01:59:03 init: getty repeating too quickly on port /dev/ttyv0, sleeping 30 secs
Sep 6 01:59:07 sshd[28230]: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:07 sshd[28230]: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:07 sshd[28230]: login_getclass: unknown class 'root'
Sep 6 01:59:07 sshd[28230]: login_getclass: unknown class 'root'
Sep 6 01:59:07 sshd[28230]: in openpam_check_desc_owner_perms(): /etc/pam.d/sshd: insecure ownership or permissions
Sep 6 01:59:07 sshd[28230]: in openpam_check_desc_owner_perms(): /etc/pam.d/sshd: insecure ownership or permissions
Sep 6 01:59:07 sshd[28230]: fatal: PAM: initialisation failed
Sep 6 01:59:07 sshd[28230]: fatal: PAM: initialisation failed

doktornotor · Sep 6, 2015, 7:07 AM

Yeah. pfSense is blocking nothing, you have messed up permissions on your system.

jolebole · Sep 6, 2015, 7:17 AM

But I have not been changing any permissions. I never changed anything from ssh since I never used it. I only have NAT and Firewall rules, DHCP, DNS and recently installed Squid. All was managed from the web interface.

Solution, clean install? Will config backup/restore all settings on the new pfinstall ?

Derelict · Sep 6, 2015, 7:51 AM

What's the output of this in Diagnostics > Command Prompt

Command: ls -l /etc

Could this be another by-product of the /etc corruption crap? Is this a nano install?

doktornotor · Sep 6, 2015, 8:40 AM

Well, whatever it is, I'd re-apply latest 2.2.4 upgrade to get same permissions everywhere.

firewalluser · Sep 6, 2015, 11:29 AM

@jolebole:

This is the system log:

Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root

[snip]

Sep 6 01:59:07 sshd[28230]: login_getclass: unknown class 'root'
Sep 6 01:59:07 sshd[28230]: login_getclass: unknown class 'root'
Sep 6 01:59:07 sshd[28230]: in openpam_check_desc_owner_perms(): /etc/pam.d/sshd: insecure ownership or permissions
Sep 6 01:59:07 sshd[28230]: in openpam_check_desc_owner_perms(): /etc/pam.d/sshd: insecure ownership or permissions
Sep 6 01:59:07 sshd[28230]: fatal: PAM: initialisation failed
Sep 6 01:59:07 sshd[28230]: fatal: PAM: initialisation failed

What do you have installed package wise and have you made any changes of your own at the command line?

You might want to look at this thread https://forum.pfsense.org/index.php?topic=92620.0

Where the "/etc/login.conf is not owned by root" message pops up, its related to the Jail which controls the permissions.

Now a possible quick and simple solution would be to use "chown root /etc/login*" but as you dont know what caused this problem, I'd investigate it first.

One thing I will say, if you have had your pfsense fw compromised and its not an unreported bug, all your devices that can be updated have the potential for being compromised, ie your bios, if using windows on a spin disk, where windows does a quick format, that can leave malware on the sectors of little used parts of the spin disk waiting to be reactivated even though windows follows an algo to split programs and files across the spin disk to speed up the user experience, printers, scanners etc where the firmware can also be updated. Its very easy decompiling code in an automated fashion and its quite easy auditing a network and systems for HW as the NSA would like to testify but cant due to their inherent requirement for operational secrecy, but they do use Linux alot. ;)

Dont under estimate the ingenuity of hackers.

What I'd be curious to know at this stage is if your pfsense machine has been compromised/hacked and if its being used in much the same way Lizard Squad have pwn'ed routers making them part of a botnet as you can read here.
http://krebsonsecurity.com/2015/01/lizard-stresser-runs-on-hacked-home-routers/

If you just re-apply the 2.2.4 patch, you will be none the wiser to finding out if your pfsense fw has been hacked or not and if it has been hacked, then you still wont be able to stop the hack until you find how they hacked your pfsense fw. Do you see the problem?

So if you want to find out, which will help all the pfsense users, you could start by making an image, make a backup of your pfsense fw before reinstalling 2.2.4 or the 2.2.2 version thats got permission issues and then start comparing your dodgy image against the newly installed image.

If you upgraded to 2.2.2 from 2.2.1 and 2.2.1 was a fresh install, I'd reinstall 2.2.1, restore that backup before upgrading to 2.2.2. This way any upgrade errors/anomalies will likely be reproduced which might also have contributed to you permissions failure. It takes about 10mins to do a fresh install, restore a backup and about another 10mins to upgrade and restore the backup for that version. It also pays to keep copies of old backups before you upgraded in case you ever have to do some sort of forensic analysis like this.

dd or dcfldd will help you do images on a linux box, then to compare cmp using a command like "cmp /dev/hda /dev/hdb" will report the differences. This will report differences as your logs will be different, but if pfsense stores logs on a seperate partition (havent looked) then it will help reduce the number of errors reported as file date time stamps will be different and will get reported the most, but its the differences file sizes you want to look for as well as additional files that will help you find out if you have you any additional code added to your fw. If that shows nothing up, the other possibility is a zero day in some code used by pfsense which may not have been reported to FreeBSD amongst a few other possibilitys.

Of course what makes this exercise even harder is the packages you may have been running, might have been updated after you original installed them. One way around this problem is to make your own offline copy of the package repository to keep copies of what you installed at the time for such a forensic exercise. This link will help you with that problem. https://doc.pfsense.org/index.php/Creating_a_Custom_Package_Repository

Anyway just a bit of food for thought if you are so inclined to find out a little more about whether your pfsense box has been hacked or not. :)

fwiw.

doktornotor · Sep 6, 2015, 12:32 PM

@firewalluser:

I'd reinstall 2.2.1, restore that backup before upgrading to 2.2.2.

Leaving the loads of OT crap aside, why on earth would you be restoring known buggy versions that caused the huge /etc corruption issues? WTF really.

jolebole · Sep 6, 2015, 3:23 PM

@doktornotor:

Well, whatever it is, I'd re-apply latest 2.2.4 upgrade to get same permissions everywhere.

I'm gonna run the upgrade and see if that helps. If not I'll do a clean install.

jolebole · Sep 6, 2015, 3:25 PM

@Derelict:

What's the output of this in Diagnostics > Command Prompt

Command: ls -l /etc

Could this be another by-product of the /etc corruption crap? Is this a nano install?

Sorry I ran the upgrade and did not check that command. It's a regular install to a hdd.

jolebole · Sep 6, 2015, 3:27 PM

@firewalluser:

@jolebole:

This is the system log:

Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root
Sep 6 01:59:03 init: _secure_path: /etc/login.conf is not owned by root

[snip]

Sep 6 01:59:07 sshd[28230]: login_getclass: unknown class 'root'
Sep 6 01:59:07 sshd[28230]: login_getclass: unknown class 'root'
Sep 6 01:59:07 sshd[28230]: in openpam_check_desc_owner_perms(): /etc/pam.d/sshd: insecure ownership or permissions
Sep 6 01:59:07 sshd[28230]: in openpam_check_desc_owner_perms(): /etc/pam.d/sshd: insecure ownership or permissions
Sep 6 01:59:07 sshd[28230]: fatal: PAM: initialisation failed
Sep 6 01:59:07 sshd[28230]: fatal: PAM: initialisation failed

What do you have installed package wise and have you made any changes of your own at the command line?

You might want to look at this thread https://forum.pfsense.org/index.php?topic=92620.0

Where the "/etc/login.conf is not owned by root" message pops up, its related to the Jail which controls the permissions.

Now a possible quick and simple solution would be to use "chown root /etc/login*" but as you dont know what caused this problem, I'd investigate it first.

One thing I will say, if you have had your pfsense fw compromised and its not an unreported bug, all your devices that can be updated have the potential for being compromised, ie your bios, if using windows on a spin disk, where windows does a quick format, that can leave malware on the sectors of little used parts of the spin disk waiting to be reactivated even though windows follows an algo to split programs and files across the spin disk to speed up the user experience, printers, scanners etc where the firmware can also be updated. Its very easy decompiling code in an automated fashion and its quite easy auditing a network and systems for HW as the NSA would like to testify but cant due to their inherent requirement for operational secrecy, but they do use Linux alot. ;)

Dont under estimate the ingenuity of hackers.

What I'd be curious to know at this stage is if your pfsense machine has been compromised/hacked and if its being used in much the same way Lizard Squad have pwn'ed routers making them part of a botnet as you can read here.
http://krebsonsecurity.com/2015/01/lizard-stresser-runs-on-hacked-home-routers/

If you just re-apply the 2.2.4 patch, you will be none the wiser to finding out if your pfsense fw has been hacked or not and if it has been hacked, then you still wont be able to stop the hack until you find how they hacked your pfsense fw. Do you see the problem?

So if you want to find out, which will help all the pfsense users, you could start by making an image, make a backup of your pfsense fw before reinstalling 2.2.4 or the 2.2.2 version thats got permission issues and then start comparing your dodgy image against the newly installed image.

If you upgraded to 2.2.2 from 2.2.1 and 2.2.1 was a fresh install, I'd reinstall 2.2.1, restore that backup before upgrading to 2.2.2. This way any upgrade errors/anomalies will likely be reproduced which might also have contributed to you permissions failure. It takes about 10mins to do a fresh install, restore a backup and about another 10mins to upgrade and restore the backup for that version. It also pays to keep copies of old backups before you upgraded in case you ever have to do some sort of forensic analysis like this.

dd or dcfldd will help you do images on a linux box, then to compare cmp using a command like "cmp /dev/hda /dev/hdb" will report the differences. This will report differences as your logs will be different, but if pfsense stores logs on a seperate partition (havent looked) then it will help reduce the number of errors reported as file date time stamps will be different and will get reported the most, but its the differences file sizes you want to look for as well as additional files that will help you find out if you have you any additional code added to your fw. If that shows nothing up, the other possibility is a zero day in some code used by pfsense which may not have been reported to FreeBSD amongst a few other possibilitys.

Of course what makes this exercise even harder is the packages you may have been running, might have been updated after you original installed them. One way around this problem is to make your own offline copy of the package repository to keep copies of what you installed at the time for such a forensic exercise. This link will help you with that problem. https://doc.pfsense.org/index.php/Creating_a_Custom_Package_Repository

Anyway just a bit of food for thought if you are so inclined to find out a little more about whether your pfsense box has been hacked or not. :)

fwiw.

I dont know if the pfsense has been compromised. I have not noticed any weird network or bandwith issues. As a precaution I will be changing the password and lock the box from the outside.

firewalluser · Sep 6, 2015, 6:34 PM

@jolebole:

I dont know if the pfsense has been compromised. I have not noticed any weird network or bandwith issues. As a precaution I will be changing the password and lock the box from the outside.

You wont or shouldnt notice anything unusual by getting at the first node in a network, ie the router or firewall, thats the point of going for these targets.

firewalluser · Sep 6, 2015, 6:44 PM

@doktornotor:

@firewalluser:

I'd reinstall 2.2.1, restore that backup before upgrading to 2.2.2.

Leaving the loads of OT crap aside, why on earth would you be restoring known buggy versions that caused the huge /etc corruption issues? WTF really.

WTF Really?

@firewalluser:

If you upgraded to 2.2.2 from 2.2.1 and 2.2.1 was a fresh install, I'd reinstall 2.2.1, restore that backup before upgrading to 2.2.2. This way any upgrade errors/anomalies will likely be reproduced which might also have contributed to you permissions failure.

All releases have bugs, including the current version 2.2.4 which you are recommending people to upgrade to, these bugs are currently unknown bugs or zero days, until reported and are patched in 2.2.5 or later versions.

So get over the fact thats the name of the game, its a moving target. Its what makes or breaks sloppy firewalls and internet security practices leaving users exposed. :D