Strange SSH issue (keys)

creiss

Hey folks,

I am running 2.3.4 happilly and just now encountered a strange behavior: My non-admin users cant login via SSH. admin user has a set of ssh keys, the admin group has shell acces permission and admin can log in.

Copying over the exact same ssh keys from admin to a user and trying to login with that user fails with permission denied.
Enabling password login (ssh) yields in successful login of a user with password.

Again, authorized_keys, clients, ips et all remain the very same. Even deleting all but one auth key will still not allow login.

ssh -vvvv shows:

debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/chris/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/chris/.ssh/id_dsa
[…]
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

What I figured:

Port is open (works for admin)
keys match public keys (works for admin)
Permissions are ok (works with password login)
"diff /root/.ssh/authorized_keys /home/chris/.ssh/authorized_keys" shows identical files
permissions of /home/chris/.ssh/authorized_keys are 0644
clog on system log show "sshd[35842]: Connection closed by my.ip port 45696

I am not sure what I am missing here.
-Chris.

johnpoz

so did you put the key in the user in the user manager or did you just copy it to the .ssh folder of that user?

ssh johnpoz@pfsense.local.lan
[2.4.0-BETA][johnpoz@pfsense.local.lan]/home/johnpoz:

userssshkey.png_thumb

creiss

User manager, of course.
As per documentaion I shall not touch the files :) - Only did a file based diff to exclude some weird issue uppon creating those files.

-Chris.

creiss

Bump

This issue is still alive an kicking :/

jimp

Do the users exist in /etc/passwd? Do they have proper shells there?

Does it work if you add the permissions to the user directly?

creiss

Hey,

yes they are in the passwd file, along with /bin/tcsh.
No, they can't login even with permissions removed from group and added to users.

If I enable password-login in sshd they can login (with password) just fine. It's their ssh key that gets denied.

jimp

Strange. I just tried the same thing (ssh key on user, user in admin group) and my user can login OK using the key.

Do you see anything in the system log when the user fails to auth with the key? What about on the client side if you run ssh with "-v"?

creiss

system.log on server on yields a meager

Jun 22 16:53:38 cerberus sshd[46382]: Connection closed by 84.246.124.42 port 43092 [preauth]

while the client just can't connect:

debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key:  (0x7fdd02b7c0a0),
[b]debug2: key: /home/chris/.ssh/id_rsa (0x7fdd02b6afd0),[/b]
debug2: key: /home/chris/.ssh/id_dsa ((nil)),
debug2: key: /home/chris/.ssh/id_ecdsa ((nil)),
debug2: key: /home/chris/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: 
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
[b]debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /home/chris/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply[/b]
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/chris/.ssh/id_dsa
debug3: no such identity: /home/chris/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/chris/.ssh/id_ecdsa
debug3: no such identity: /home/chris/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/chris/.ssh/id_ed25519
debug3: no such identity: /home/chris/.ssh/id_ed25519: No such file or directory
[b]debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).[/b]

Again: Same key, same account just pasting the key into the admin ssh box will make the login successful.

jimp

Is that an older key? It may be too weak for current standards. Though I'm not sure why it would work for root and not a user account.

Can you try making and using a new key like an ed25519 key and see if that works?

creiss

It's a 4096bit rsa key, but sure.

creiss

debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/chris/.ssh/id_dsa
debug3: no such identity: /home/chris/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/chris/.ssh/id_ecdsa
debug3: no such identity: /home/chris/.ssh/id_ecdsa: No such file or directory
debug1: Offering ED25519 public key: /home/chris/.ssh/id_ed25519
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

jimp

And nothing in the server log except the connection being closed?

An RSA key should still work. My output looks similar but it changes around here:

debug1: Offering RSA public key: /home/jim/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg rsa-sha2-512 blen 277

Which is why I find it odd that the server is rejecting it without logging exactly why.

Are you sure you are hitting this firewall and not getting port forwarded to a different firewall or system?

creiss

I'd say, yes:

~/.ssh> ssh -v admin@10.4.0.1
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 10.4.0.1 [10.4.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/chris/.ssh/id_rsa type 1
debug1: identity file /home/chris/.ssh/id_rsa-cert type -1
debug1: identity file /home/chris/.ssh/id_dsa type -1
debug1: identity file /home/chris/.ssh/id_dsa-cert type -1
debug1: identity file /home/chris/.ssh/id_ecdsa type -1
debug1: identity file /home/chris/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/chris/.ssh/id_ed25519 type 4
debug1: identity file /home/chris/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2
debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
debug1: kex: server->client aes128-ctr umac-128-etm@openssh.com zlib@openssh.com
debug1: kex: client->server aes128-ctr umac-128-etm@openssh.com zlib@openssh.com
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ED25519 28:c4:e9:25:ad:c2:74:8b:32:99:f9:81:3c:8d:78:d4
debug1: skipped DNS lookup for numerical hostname
debug1: Host '10.4.0.1' is known and matches the ED25519 host key.
debug1: Found key in /home/chris/.ssh/known_hosts:6
debug1: ssh_ed25519_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: 
debug1: Server accepts key: pkalg ssh-rsa blen 535
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).

~/.ssh> ssh -v chris@10.4.0.1
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 10.4.0.1 [10.4.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/chris/.ssh/id_rsa type 1
debug1: identity file /home/chris/.ssh/id_rsa-cert type -1
debug1: identity file /home/chris/.ssh/id_dsa type -1
debug1: identity file /home/chris/.ssh/id_dsa-cert type -1
debug1: identity file /home/chris/.ssh/id_ecdsa type -1
debug1: identity file /home/chris/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/chris/.ssh/id_ed25519 type 4
debug1: identity file /home/chris/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2
debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-128-etm@openssh.com zlib@openssh.com
debug1: kex: client->server aes128-ctr umac-128-etm@openssh.com zlib@openssh.com
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ED25519 28:c4:e9:25:ad:c2:74:8b:32:99:f9:81:3c:8d:78:d4
debug1: skipped DNS lookup for numerical hostname
debug1: Host '10.4.0.1' is known and matches the ED25519 host key.
debug1: Found key in /home/chris/.ssh/known_hosts:6
debug1: ssh_ed25519_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: 
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /home/chris/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/chris/.ssh/id_dsa
debug1: Trying private key: /home/chris/.ssh/id_ecdsa
debug1: Offering ED25519 public key: /home/chris/.ssh/id_ed25519
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

Same ip (both ends), Same originating account, same SSH Keys, same copy&pasted allowed ssh block for admin & chris. One joy one nay.

jimp

Hmm, ok. And I did confirm that a key mismatch only prints the connection closed message in the logs.

So somehow it's either not able to read those keys or it believes the accounts are not valid, neither of which make much sense given the context.

We don't have any way to increase the sshd logging in the GUI, but you could manually edit the /etc/sshd script and add a line to it so it puts "LogLevel DEBUG2" in the config, and then run /etc/sshd to restart sshd and see what shows up.

creiss

Here goes:

Jun 22 17:23:55 cerberus sshd[88455]: Connection from XX port 43848 on 10.4.0.1 port 22
Jun 22 17:23:55 cerberus sshd[88455]: Failed publickey for chris from XX port 43848 ssh2: RSA SHA256:/uRgUtSvDjHkSiSPJG1tDjbSzBwhocphMLgm5VvVH/Q
Jun 22 17:23:55 cerberus sshd[88455]: Failed publickey for chris from XX port 43848 ssh2: RSA SHA256:K6/VRlhI/MHe+EwjMx1gT5WphKM/LLDbZe+mQzBQBE4
Jun 22 17:23:55 cerberus sshd[88455]: Failed publickey for chris from XX port 43848 ssh2: ED25519 SHA256:lIR6M20L0fvmWqzapIDz6Q2FcNuSDybJv9NXF8P86hY
Jun 22 17:23:55 cerberus sshd[88455]: Connection closed by XX port 43848 [preauth]

cat /home/chris/.ssh/authorized_keys 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHegX2pTGXjLB5YBIVa/x3IfZ6f26idLd5pZ3WIffJni chris@kerberos.alpha-labs.net

Humm :/

jimp

How many entries are in that user's authorized_keys file? Three? It is strange that it is seeing them but deciding they don't match.

There must be something even more simple/basic at play here that is getting overlooked. Are you sure the authorized key data was pasted in properly, without newlines in incorrect places?

Can you copy and paste it into a text editor and confirm there is one whole key per line?

creiss

Hey,

Yeah. I wiped, redid, saved, copied the block out again into a text editor: One key per line.
If i take the old key block of the user and copy&paste it as admin I can log in as admin easily.

johnpoz

I can not duplicate this.. I use the same key for my user as my root/admin account. I create a new user, copy the key in and bam in like flint.


> ssh -v testssh@pfsense.local.lan
OpenSSH_7.5p1, OpenSSL 1.0.2k  26 Jan 2017
debug1: Reading configuration data /home/jpoz/.ssh/config
debug1: Connecting to pfsense.local.lan [192.168.9.253] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/jpoz/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jpoz/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jpoz/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jpoz/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jpoz/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jpoz/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/jpoz/.ssh/id_ed25519 type 4
debug1: key_load_public: No such file or directory
debug1: identity file /home/jpoz/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2
debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to pfsense.local.lan:22 as 'testssh'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit>compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit>compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-ed25519 SHA256:I0WQR9Eyjlcgf/vNFHQ7sm290r/MV+3tQ8QZfxEk9pw
debug1: Host 'pfsense.local.lan' is known and matches the ED25519 host key.
debug1: Found key in /home/jpoz/.ssh/known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs= <rsa-sha2-256,rsa-sha2-512>debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/jpoz/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/jpoz/.ssh/id_dsa
debug1: Trying private key: /home/jpoz/.ssh/id_ecdsa
debug1: Offering ED25519 public key: /home/jpoz/.ssh/id_ed25519
debug1: Server accepts key: pkalg ssh-ed25519 blen 51
debug1: Authentication succeeded (publickey).
Authenticated to pfsense.local.lan ([192.168.9.253]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
[2.4.0-BETA][testssh@pfsense.local.lan]/home/testssh:</rsa-sha2-256,rsa-sha2-512></implicit></implicit>

creiss

Hey folks,

I found the root cause. I initially installed pfsense then restored a configuration. I can only assume that this created the user home dirs in the first place. During the adjustments needed I decided to start over, so I reset the configuarion and re-did everything.

The home directories are owned by the other users, for example:

user1 is owned by user2
user2 is owned by user3
user3 is owned by user1

I can only assume that the uid mapping changed due to re-creating the users while not wiping the home directories. Seems like the revert option does not wipe /home.

Ah well, another mystery solved.