SSH Permission Denied from only one client

justynnuff · Nov 3, 2021, 7:57 PM

Hopefully this is the right place to post this question.

I have a small home networking setup with a few virtualized machines running in Proxmox. I created and added the RSA public keys to the PfSense interface under System -> User Manager -> Users -> admin (edit) -> Keys -> Authorized SSH Keys for each of my guest machines, and everything works perfectly fine. Each new machine's public key is added to Authorized SSH Keys on a new line, and they're all 2048 bits in length.

However, when I add the key for my proxmox host (also RSA 2048), and try to ssh in, I get

root@192.168.1.1: Permission denied (publickey).

In the pfSense logs, I see

192.168.1.201 is the Proxmox host.

This is the output from Proxmox when I try to ssh into pfsense:

root@proxmox:~# ssh -vvv root@192.168.1.1
OpenSSH_8.4p1 Debian-5, OpenSSL 1.1.1k  25 Mar 2021
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.1 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.1 [192.168.1.1] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Debian-5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9
debug1: match: OpenSSH_7.9 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.168.1.1:22 as 'root'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ED25519 in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 192.168.1.1
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes128-ctr MAC: umac-128-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-128-etm@openssh.com compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-ed25519 SHA256:Xl/x+jReWHCJjHPuLVd6slXufRZ9dLwodrSg9kS2zsA
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ED25519 in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 192.168.1.1
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: Host '192.168.1.1' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /root/.ssh/id_rsa RSA SHA256:nKGaxU7ZV8OTrkrkGJEmjJ7WW6t88aWkorW2ZY2Seck
debug1: Will attempt key: /root/.ssh/id_dsa 
debug1: Will attempt key: /root/.ssh/id_ecdsa 
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /root/.ssh/id_ed25519 
debug1: Will attempt key: /root/.ssh/id_ed25519_sk 
debug1: Will attempt key: /root/.ssh/id_xmss 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa RSA SHA256:nKGaxU7ZV8OTrkrkGJEmjJ7WW6t88aWkorW2ZY2Seck
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa_sk
debug3: no such identity: /root/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519_sk
debug3: no such identity: /root/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /root/.ssh/id_xmss
debug3: no such identity: /root/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
root@192.168.1.1: Permission denied (publickey).

The difference between the verbose ssh output between the guests that work, and Proxmox, which doesn't, starts at

debug3: receive packet: type 51

On the working machines, the output at that point is

debug3: receive packet: type 60

But I'm not sure what that difference means after googling around.

Is there any obvious stupid thing I'm doing? Is it because the username on Proxmox is root instead of my usual username, or something similar? I would assume on the client side, that doesn't matter, right?

*Also I forgot to mention that this used to work up until recently. I have cronjobs on all the machines to grab the keys from pfSense's ACME certs folder and pull them in locally so I can have a split-DNS configuration where everything has actual, not self-signed, keys. Proxmox reported the error trying to scp the keys a couple nights ago. So did the Proxmox box look suspicious to pfSense, and pfSense locked it out permanently? All the other machines are totally fine running the same cronjobs.

stephenw10 · Nov 3, 2021, 8:56 PM

It's the wrong key?

I can't replicate that here. I just tried adding the public key from PVE to pfSense and it worked as expected.

I tested from Proxmox 6.4-13 into pfSense 22.01 but I don't imagine any issue with other versions.

Steve

justynnuff · Nov 3, 2021, 9:06 PM

@stephenw10 I'm taking the key from /root/.ssh/id_rsa.pub on Proxmox, so unless I should be grabbing the public key from somewhere else, I don't think that's it. I've triple-double checked, and even generated keys of different sizes, settling on 2048 because it matched my other VMs that DID work, although that wasn't the problem. Every time I made sure the keys matched in the pfSense UI and PVE id_rsa.pub file.

It feels like pfSense has locked out PVE on its end for whatever reason, but I can't figure out how to diagnose the issue any further.

stephenw10 · Nov 3, 2021, 9:33 PM

Mmm, pfSense is refusing the key you are sending. It's happy with what I'm sending:

debug1: Offering public key: /root/.ssh/id_rsa RSA SHA256:B5ZSTiGsin8iddb4SjX3Nr0iZc1c3jYyS9wfygoHDTY
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /root/.ssh/id_rsa RSA SHA256:B5ZSTiGsin8iddb4SjX3Nr0iZc1c3jYyS9wfygoHDTY
debug3: sign_and_send_pubkey: RSA SHA256:B5ZSTiGsin8iddb4SjX3Nr0iZc1c3jYyS9wfygoHDTY
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).

The only significant difference I can see there is the SSH version. Yours is a lot newer:

root@pve:~/.ssh# ssh -vvv admin@6100.stevew.lan
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019

I wonder if it's sending something newer than the SSH in pfSense can use. I don't see that in the logs though.

Can you connect using a password if it's enabled?

Steve

justynnuff · Nov 3, 2021, 10:48 PM

@stephenw10 Excellent suggestion. I enabled "Password or public key" in the System -> Advanced -> Secure Shell options, and I was able to login from my Proxmox box.

It doesn't like my keys?

(at the risk of showing my RSA keys)

stephenw10 · Nov 4, 2021, 5:12 AM

Hmm, nothing obviously wrong...

I'd probably try to start sshd with debug logging enabled in pfSense and see what errors it throws.

Steve

justynnuff · Nov 4, 2021, 5:13 AM

I turned on debugging in sshd and it looked like it wasn't able to find my keys in the authorized_keys file on the pfSense box, even though they were there. Long stupid story short, there were carriage returns in my ssh keys when I copy and pasted them over from Cygwin.

I could see them in 'vi' on the pfSense box as ^M's in the authorized_keys file....

Thank you for your help @stephenw10. Hopefully this might help someone in the distant future.