SSHD failed to start

[KrypticKahos]

@stephenw10

After reboot or stopping and attempting to restart sshd ps -auxwwd gives no reference to sshd. After running /usr/sbin/sshd I do see

root    86270   0.0  0.1  19568   9128  -  Ss   02:22      0:00.00 |-- /usr/sbin/sshd

but no other sub-processes.

I haven't made any modifications to the crypto settings.

[stephenw10]

Hmm, so it does it at anytime it's restarted even when the keys already exist?

[KrypticKahos]

@stephenw10

That is correct.

The only way I am able to get it to start is by running the sbin command, and it gives me a missing key error but then sshd starts up and works.

I’m unable to start sshd using VGA terminal inputs, via the services widget, or automatically when pfsense restarts.

[stephenw10]

What's the exact error it shows?

[KrypticKahos]

@stephenw10

Could not load host key: /etc/ssh/ssh_host_rsa_key

[manicmoose]

@kryptickahos

I had the same SSHD issue with clean installs of 2.6.0, as mentioned here.

Looking at your "ls" output earlier, you don't have an 'ssh_host_rsa_key' - which is what 2.6.0 seems it now looks for - so create one:

cd /etc/ssh
ssh-keygen -N '' -t rsa -f ssh_host_rsa_key
/usr/sbin/sshd

...try a reboot after that and it should now be fixed.

[KrypticKahos]

@manicmoose
While a good suggestion, I attempted it and have the same issue after reboot. It did generate the key, but after restart sshd didn't start, and running /usr/sbin/sshd still tells me the key is missing.

Also as a side note I have been having this issue since performing a clean install of 2.5.2

[manicmoose]

@kryptickahos What are the actual contents of your sshd config?

cat /etc/ssh/sshd_config

[KrypticKahos]

@manicmoose

# This file is automatically generated at startup
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
Compression delayed
ClientAliveInterval 30
PermitRootLogin yes
# Login via Key or Password
ChallengeResponseAuthentication yes
PasswordAuthentication yes
PubkeyAuthentication yes
UseDNS no
LoginGraceTime 30s
VersionAddendum none
AllowAgentForwarding no
X11Forwarding no
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
# override default of no subsystems
Subsystem	sftp	/usr/libexec/sftp-server

[manicmoose]

@kryptickahos
Okay - so that looks identical to mine so should be fine.
The only difference is that I'm running 'CA' 2.6.0, not 22.01+

I doubt that would matter, so it's a bit of a strange one.
Now that you've generated the 'rsa' key, does the error still say the same thing, ie.
Could not load host key: /etc/ssh/ssh_host_rsa_key ?

[KrypticKahos]

@manicmoose
Since then I have reverted back to CE on 2.6.0
And yes, after generating ssh_host_rsa_key I still get the error when starting sshd.

[manicmoose]

@kryptickahos
Well, it's going to be a strange outcome.

If you've got the file that pfSense is whinging about and the perm's are all correct, then there's something probably only Netgate can answer going wrong.
If you can't get any joy here then I guess you'll have to re-install from scratch and not restore your config. Good luck...!

[KrypticKahos]

I managed to get sshd working as expected. Failure for the keys to be generated correctly appears have been related to expiration of webConfigurator default certificate.

I renewed the self signed cert, disable ssh, rebooted, re-enabled ssh. Received the keys are being generated notification and then immediately keys were generated and now sshd works correctly.

[manicmoose]

@kryptickahos
Hmmm....thought the expired cert would have come up as a pretty obvious alert on the web console page.
Either way, it would have been nice to see this error also in the logs to give you a clue.
Nice you've sorted it, anyhow!!

[KrypticKahos]

@manicmoose
It was obvious that the cert was expired, just didn't think it would have anything to do with ssh key generation. I also incorrectly assumed it was from an old cert when I was playing with the acme plugin.

It's still weird I was able to start ssh from command prompt, and it gave a the missing key error, but either way I'm glad its working now.

[stephenw10]

Mmm, that is weird. I can't really see how an expired cert would affect that. I'll see if I can replicate it.

Steve

[KrypticKahos]

@stephenw10
One final piece of information that may help with the actual root cause of the issue. When troubleshooting I also found an old static IP entry on LAN for the device I was using as my firewall. This was an old entry from when I was using a different firewall and had this device on the network. During the time of getting the fix working I also deleted the static IP entry, but it didn't immediately resolve the issue so I didn't think it was the solution but it might have been.

I attempted further testing by restoring to the old config that had the static IP as well as the expired self signed cert (I didn't perform a clean install this time). At reboot after restoring with both issues sshd worked correctly.

So this may actually be related to the static IP entry but only manifests after a clean install and restore. If I get some time in the future I may attempt a clean install again but as of current this is the best info I have.

As an extra point the firewall I'm using is a dual Ethernet port device, with non-switched ports. I'm not sure if the WAN or LAN port was assigned the static IP, but it was one of them.

[lxndrp]

@stephenw10 @manicmoose

I have the impression that we have several issues that somehow interact here.

First, it seems that (for our systems) after an upgrade from 2.5.2 to 2.6, SSH host keys get lost. I have verified this for my installation (with a clean install) already.

Second, a re-install from USB (with config.xml recovery from disk) has the same issue. I have verified this as well: A manual backup of the config.xml before doing the reinstall has the keys; after the reinstall, they seem to be lost.

Third, although keys are in place (e.g. by running

ssh-keygen -A

from the Diagnostics/Command Prompt), sshd fails to start (logs look ok, Status/Services indicates a red X, running

ssh admin@pfsense

against the pfSense host times out.

However, when running

/usr/sbin/sshd

from the Diagnostics/Command Prompt manually, it seems to start fine (no output at all), and sshing into the machine works fine.

After rebooting a few times (usually once doesn't suffice), sshd seems to start normally.

Let me know if I can provide you with additional information (logs etc.); I am happy to help.

[stephenw10]

Hmm, ssh keys certainly shouldn't be lost at upgrade.

Storing of SSH keys in the config is new in 22.01/2.6 so if you are restoring a backup from 2.5.2 the keys would not be restored.

You see any errors when restoring a 2.6 config into 2.6?

Steve

[lxndrp]

@stephenw10 I only tried from 22.01 to 2.6.0; there, the same issue appears.