Strange SSH issue (keys)



  • Hey folks,

    I am running 2.3.4 happilly and just now encountered a strange behavior: My non-admin users cant login via SSH. admin user has a set of ssh keys, the admin group has shell acces permission and admin can log in.

    Copying over the exact same ssh keys from admin to a user and trying to login with that user fails with permission denied.
    Enabling password login (ssh) yields in successful login of a user with password.

    Again, authorized_keys, clients, ips et all remain the very same. Even deleting all but one auth key will still not allow login.

    ssh -vvvv shows:

    debug3: remaining preferred: keyboard-interactive,password
    debug3: authmethod_is_enabled publickey
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: /home/chris/.ssh/id_rsa
    debug3: send_pubkey_test
    debug2: we sent a publickey packet, wait for reply
    debug1: Authentications that can continue: publickey
    debug1: Trying private key: /home/chris/.ssh/id_dsa
    […]
    debug2: we did not send a packet, disable method
    debug1: No more authentication methods to try.
    Permission denied (publickey).

    What I figured:

    • Port is open (works for admin)
    • keys match public keys (works for admin)
    • Permissions are ok (works with password login)
    • "diff /root/.ssh/authorized_keys /home/chris/.ssh/authorized_keys" shows identical files
    • permissions of /home/chris/.ssh/authorized_keys are 0644
    • clog on system log show "sshd[35842]: Connection closed by my.ip port 45696

    I am not sure what I am missing here.
    -Chris.


  • Rebel Alliance Global Moderator

    so did you put the key in the user in the user manager or did you just copy it to the .ssh folder of that user?

    ssh johnpoz@pfsense.local.lan
    [2.4.0-BETA][johnpoz@pfsense.local.lan]/home/johnpoz:




  • User manager, of course.
    As per documentaion I shall not touch the files :) - Only did a file based diff to exclude some weird issue uppon creating those files.

    -Chris.



  • Bump

    This issue is still alive an kicking :/


  • Rebel Alliance Developer Netgate

    Do the users exist in /etc/passwd? Do they have proper shells there?

    Does it work if you add the permissions to the user directly?



  • Hey,

    yes they are in the passwd file, along with /bin/tcsh.
    No, they can't login even with permissions removed from group and added to users.

    If I enable password-login in sshd they can login (with password) just fine. It's their ssh key that gets denied.


  • Rebel Alliance Developer Netgate

    Strange. I just tried the same thing (ssh key on user, user in admin group) and my user can login OK using the key.

    Do you see anything in the system log when the user fails to auth with the key? What about on the client side if you run ssh with "-v"?



  • system.log on server on yields a meager

    Jun 22 16:53:38 cerberus sshd[46382]: Connection closed by 84.246.124.42 port 43092 [preauth]
    

    while the client just can't  connect:

    debug2: service_accept: ssh-userauth
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug2: key:  (0x7fdd02b7c0a0),
    [b]debug2: key: /home/chris/.ssh/id_rsa (0x7fdd02b6afd0),[/b]
    debug2: key: /home/chris/.ssh/id_dsa ((nil)),
    debug2: key: /home/chris/.ssh/id_ecdsa ((nil)),
    debug2: key: /home/chris/.ssh/id_ed25519 ((nil)),
    debug1: Authentications that can continue: publickey
    debug3: start over, passed a different list publickey
    debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
    debug3: authmethod_lookup publickey
    debug3: remaining preferred: keyboard-interactive,password
    debug3: authmethod_is_enabled publickey
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: 
    debug3: send_pubkey_test
    debug2: we sent a publickey packet, wait for reply
    [b]debug1: Authentications that can continue: publickey
    debug1: Offering RSA public key: /home/chris/.ssh/id_rsa
    debug3: send_pubkey_test
    debug2: we sent a publickey packet, wait for reply[/b]
    debug1: Authentications that can continue: publickey
    debug1: Trying private key: /home/chris/.ssh/id_dsa
    debug3: no such identity: /home/chris/.ssh/id_dsa: No such file or directory
    debug1: Trying private key: /home/chris/.ssh/id_ecdsa
    debug3: no such identity: /home/chris/.ssh/id_ecdsa: No such file or directory
    debug1: Trying private key: /home/chris/.ssh/id_ed25519
    debug3: no such identity: /home/chris/.ssh/id_ed25519: No such file or directory
    [b]debug2: we did not send a packet, disable method
    debug1: No more authentication methods to try.
    Permission denied (publickey).[/b]
    

    Again: Same key, same account just pasting the key into the admin ssh box will make the login successful.


  • Rebel Alliance Developer Netgate

    Is that an older key? It may be too weak for current standards. Though I'm not sure why it would work for root and not a user account.

    Can you try making and using a new key like an ed25519 key and see if that works?



  • It's a 4096bit rsa key, but sure.



  • debug2: we sent a publickey packet, wait for reply
    debug1: Authentications that can continue: publickey
    debug1: Trying private key: /home/chris/.ssh/id_dsa
    debug3: no such identity: /home/chris/.ssh/id_dsa: No such file or directory
    debug1: Trying private key: /home/chris/.ssh/id_ecdsa
    debug3: no such identity: /home/chris/.ssh/id_ecdsa: No such file or directory
    debug1: Offering ED25519 public key: /home/chris/.ssh/id_ed25519
    debug3: send_pubkey_test
    debug2: we sent a publickey packet, wait for reply
    debug1: Authentications that can continue: publickey
    debug2: we did not send a packet, disable method
    debug1: No more authentication methods to try.
    Permission denied (publickey).
    

  • Rebel Alliance Developer Netgate

    And nothing in the server log except the connection being closed?

    An RSA key should still work. My output looks similar but it changes around here:

    debug1: Offering RSA public key: /home/jim/.ssh/id_rsa
    debug2: we sent a publickey packet, wait for reply
    debug1: Server accepts key: pkalg rsa-sha2-512 blen 277
    
    

    Which is why I find it odd that the server is rejecting it without logging exactly why.

    Are you sure you are hitting this firewall and not getting port forwarded to a different firewall or system?



  • I'd say, yes:

    ~/.ssh> ssh -v admin@10.4.0.1
    OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: Connecting to 10.4.0.1 [10.4.0.1] port 22.
    debug1: Connection established.
    debug1: identity file /home/chris/.ssh/id_rsa type 1
    debug1: identity file /home/chris/.ssh/id_rsa-cert type -1
    debug1: identity file /home/chris/.ssh/id_dsa type -1
    debug1: identity file /home/chris/.ssh/id_dsa-cert type -1
    debug1: identity file /home/chris/.ssh/id_ecdsa type -1
    debug1: identity file /home/chris/.ssh/id_ecdsa-cert type -1
    debug1: identity file /home/chris/.ssh/id_ed25519 type 4
    debug1: identity file /home/chris/.ssh/id_ed25519-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_6.6.1
    debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2
    debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
    debug1: kex: server->client aes128-ctr umac-128-etm@openssh.com zlib@openssh.com
    debug1: kex: client->server aes128-ctr umac-128-etm@openssh.com zlib@openssh.com
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
    debug1: sending SSH2_MSG_KEX_ECDH_INIT
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug1: Server host key: ED25519 28:c4:e9:25:ad:c2:74:8b:32:99:f9:81:3c:8d:78:d4
    debug1: skipped DNS lookup for numerical hostname
    debug1: Host '10.4.0.1' is known and matches the ED25519 host key.
    debug1: Found key in /home/chris/.ssh/known_hosts:6
    debug1: ssh_ed25519_verify: signature correct
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: 
    debug1: Server accepts key: pkalg ssh-rsa blen 535
    debug1: Enabling compression at level 6.
    debug1: Authentication succeeded (publickey).
    
    ~/.ssh> ssh -v chris@10.4.0.1
    OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: Connecting to 10.4.0.1 [10.4.0.1] port 22.
    debug1: Connection established.
    debug1: identity file /home/chris/.ssh/id_rsa type 1
    debug1: identity file /home/chris/.ssh/id_rsa-cert type -1
    debug1: identity file /home/chris/.ssh/id_dsa type -1
    debug1: identity file /home/chris/.ssh/id_dsa-cert type -1
    debug1: identity file /home/chris/.ssh/id_ecdsa type -1
    debug1: identity file /home/chris/.ssh/id_ecdsa-cert type -1
    debug1: identity file /home/chris/.ssh/id_ed25519 type 4
    debug1: identity file /home/chris/.ssh/id_ed25519-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_6.6.1
    debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2
    debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug1: kex: server->client aes128-ctr umac-128-etm@openssh.com zlib@openssh.com
    debug1: kex: client->server aes128-ctr umac-128-etm@openssh.com zlib@openssh.com
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
    debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
    debug1: sending SSH2_MSG_KEX_ECDH_INIT
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug1: Server host key: ED25519 28:c4:e9:25:ad:c2:74:8b:32:99:f9:81:3c:8d:78:d4
    debug1: skipped DNS lookup for numerical hostname
    debug1: Host '10.4.0.1' is known and matches the ED25519 host key.
    debug1: Found key in /home/chris/.ssh/known_hosts:6
    debug1: ssh_ed25519_verify: signature correct
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: 
    debug1: Authentications that can continue: publickey
    debug1: Offering RSA public key: /home/chris/.ssh/id_rsa
    debug1: Authentications that can continue: publickey
    debug1: Trying private key: /home/chris/.ssh/id_dsa
    debug1: Trying private key: /home/chris/.ssh/id_ecdsa
    debug1: Offering ED25519 public key: /home/chris/.ssh/id_ed25519
    debug1: Authentications that can continue: publickey
    debug1: No more authentication methods to try.
    Permission denied (publickey).
    

    Same ip (both ends), Same originating account, same SSH Keys, same copy&pasted allowed ssh block for admin & chris. One joy one nay.


  • Rebel Alliance Developer Netgate

    Hmm, ok. And I did confirm that a key mismatch only prints the connection closed message in the logs.

    So somehow it's either not able to read those keys or it believes the accounts are not valid, neither of which make much sense given the context.

    We don't have any way to increase the sshd logging in the GUI, but you could manually edit the /etc/sshd script and add a line to it so it puts "LogLevel DEBUG2" in the config, and then run /etc/sshd to restart sshd and see what shows up.



  • Here goes:

    Jun 22 17:23:55 cerberus sshd[88455]: Connection from XX port 43848 on 10.4.0.1 port 22
    Jun 22 17:23:55 cerberus sshd[88455]: Failed publickey for chris from XX port 43848 ssh2: RSA SHA256:/uRgUtSvDjHkSiSPJG1tDjbSzBwhocphMLgm5VvVH/Q
    Jun 22 17:23:55 cerberus sshd[88455]: Failed publickey for chris from XX port 43848 ssh2: RSA SHA256:K6/VRlhI/MHe+EwjMx1gT5WphKM/LLDbZe+mQzBQBE4
    Jun 22 17:23:55 cerberus sshd[88455]: Failed publickey for chris from XX port 43848 ssh2: ED25519 SHA256:lIR6M20L0fvmWqzapIDz6Q2FcNuSDybJv9NXF8P86hY
    Jun 22 17:23:55 cerberus sshd[88455]: Connection closed by XX port 43848 [preauth]
    
    cat /home/chris/.ssh/authorized_keys 
    ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHegX2pTGXjLB5YBIVa/x3IfZ6f26idLd5pZ3WIffJni chris@kerberos.alpha-labs.net
    

    Humm :/


  • Rebel Alliance Developer Netgate

    How many entries are in that user's authorized_keys file? Three? It is strange that it is seeing them but deciding they don't match.

    There must be something even more simple/basic at play here that is getting overlooked. Are you sure the authorized key data was pasted in properly, without newlines in incorrect places?

    Can you copy and paste it into a text editor and confirm there is one whole key per line?



  • Hey,

    Yeah. I wiped, redid, saved, copied the block out again into a text editor: One key per line.
    If i take the old key block of the user and copy&paste it as admin I can log in as admin easily.


  • Rebel Alliance Global Moderator

    I can not duplicate this.. I use the same key for my user as my root/admin account.  I create a new user, copy the key in and bam in like flint.

    
    > ssh -v testssh@pfsense.local.lan
    OpenSSH_7.5p1, OpenSSL 1.0.2k  26 Jan 2017
    debug1: Reading configuration data /home/jpoz/.ssh/config
    debug1: Connecting to pfsense.local.lan [192.168.9.253] port 22.
    debug1: Connection established.
    debug1: key_load_public: No such file or directory
    debug1: identity file /home/jpoz/.ssh/id_rsa type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /home/jpoz/.ssh/id_rsa-cert type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /home/jpoz/.ssh/id_dsa type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /home/jpoz/.ssh/id_dsa-cert type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /home/jpoz/.ssh/id_ecdsa type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /home/jpoz/.ssh/id_ecdsa-cert type -1
    debug1: identity file /home/jpoz/.ssh/id_ed25519 type 4
    debug1: key_load_public: No such file or directory
    debug1: identity file /home/jpoz/.ssh/id_ed25519-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_7.5
    debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2
    debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
    debug1: Authenticating to pfsense.local.lan:22 as 'testssh'
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug1: kex: algorithm: curve25519-sha256@libssh.org
    debug1: kex: host key algorithm: ssh-ed25519
    debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit>compression: none
    debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit>compression: none
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug1: Server host key: ssh-ed25519 SHA256:I0WQR9Eyjlcgf/vNFHQ7sm290r/MV+3tQ8QZfxEk9pw
    debug1: Host 'pfsense.local.lan' is known and matches the ED25519 host key.
    debug1: Found key in /home/jpoz/.ssh/known_hosts:3
    debug1: rekey after 134217728 blocks
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: rekey after 134217728 blocks
    debug1: SSH2_MSG_EXT_INFO received
    debug1: kex_input_ext_info: server-sig-algs= <rsa-sha2-256,rsa-sha2-512>debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey
    debug1: Next authentication method: publickey
    debug1: Trying private key: /home/jpoz/.ssh/id_rsa
    debug1: Authentications that can continue: publickey
    debug1: Trying private key: /home/jpoz/.ssh/id_dsa
    debug1: Trying private key: /home/jpoz/.ssh/id_ecdsa
    debug1: Offering ED25519 public key: /home/jpoz/.ssh/id_ed25519
    debug1: Server accepts key: pkalg ssh-ed25519 blen 51
    debug1: Authentication succeeded (publickey).
    Authenticated to pfsense.local.lan ([192.168.9.253]:22).
    debug1: channel 0: new [client-session]
    debug1: Requesting no-more-sessions@openssh.com
    debug1: Entering interactive session.
    debug1: pledge: network
    debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
    [2.4.0-BETA][testssh@pfsense.local.lan]/home/testssh:</rsa-sha2-256,rsa-sha2-512></implicit></implicit> 
    


  • Hey folks,

    I found the root cause. I initially installed pfsense then restored a configuration. I can only assume that this created the user home dirs in the first place. During the adjustments needed I decided to start over, so I reset the configuarion and re-did everything.

    The home directories are owned by the other users, for example:

    user1 is owned by user2
    user2 is owned by user3
    user3 is owned by user1

    I can only assume that the uid mapping changed due to re-creating the users while not wiping the home directories. Seems like the revert option does not wipe /home.

    Ah well, another mystery solved.