[Solved] SSHD not starting after upgrade to 2.2

cellobita

I have successfully upgraded about twenty pfSense installs, from 2.1.5 to 2.2; on a couple of them, however, the SSHD service fails to start, and SSH access is therefore unavailable.

I have tried starting the service via the GUI, to no avail; manually executing /usr/sbin/sshd results in the following error messages:

Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Could not load host key: /etc/ssh/ssh_host_ed25519_key

Any ideas?

TIA

Marcello
São Paulo - SP - Brazil

NOYB

I also had some trouble getting SSH to work.

https://forum.pfsense.org/index.php?topic=87548.0

Does the console option 14 say to enable or disable ssh?

If you run console option 14 does it change option from one to the other? Mine would not. Always stayed on enable ssh. But would prompt as though it was already enabled and ask to disable. And visa-versa. Like something was out of sync and the toggle action would turn something on and something else off.

Then enabling it in web gui worked for me. So your issue sounds a little different.

Derelict

Do those files exist and do they look sane? Should contain:

–---BEGIN EC PRIVATE KEY-----
blah blah blah
-----END EC PRIVATE KEY-----

and

-----BEGIN OPENSSH PRIVATE KEY-----
blah blah blah
-----END OPENSSH PRIVATE KEY-----

(don't post yours. :/)

I think they're created when you upgrade to 2.2, or when sshd first starts after upgrading.

I don't know what the trigger is to make them regenerate. If they exist and contain bad keys, I'd delete them and their corresponding .pub files and see if starting sshd creates new ones.

cellobita

They do not exist:

$ ls -ahl /etc/ssh/
total 560
drwxr-xr-x   2 root  wheel   512B Jan 29 00:46 .
drwxr-xr-x  29 root  wheel   4.5K Jan 29 00:48 ..
-rw-r--r--   1 root  wheel   236K Jan 22 19:07 moduli
-rw-r--r--   1 root  wheel   1.6K Jan 22 19:07 ssh_config
-rw-------   1 root  wheel   672B Jul 27  2013 ssh_host_dsa_key
-rw-r--r--   1 root  wheel   613B Jul 27  2013 ssh_host_dsa_key.pub
-rw-------   1 root  wheel   988B Jul 27  2013 ssh_host_key
-rw-r--r--   1 root  wheel   653B Jul 27  2013 ssh_host_key.pub
-rw-------   1 root  wheel   1.6K Jul 27  2013 ssh_host_rsa_key
-rw-r--r--   1 root  wheel   405B Jul 27  2013 ssh_host_rsa_key.pub
-rw-r--r--   1 root  wheel   436B Jan 29 01:03 sshd_config

I've checked on another system that upgraded without issues, and they are there.

Disabling and reenabling Secure Shell via the GUI doesn't work.

Guess my next question is, how do I (re)generate those files without reinstalling?

Derelict

I know you can just gen them with the right ssh-keygen commands but there's probably something you can do that will make them regenerate automatically. I don't know what that is.

You can try this in the shell:

cd /etc/ssh
ssh-keygen -N '' -t ecdsa -f ssh_host_ecdsa_key
ssh-keygen -N '' -t ed25519 -f ssh_host_ed25519_key

'' is two single quotes indicating an empty passphrase.

Then start sshd and see what happens.

cellobita

It worked! I had to use the command prompt via the GUI (I'm far from the customer), and so had to concatenate the command into a single line, but this did the trick:

cd /etc/ssh ; ssh-keygen -N '' -t ecdsa -f ssh_host_ecdsa_key ; ssh-keygen -N '' -t ed25519 -f ssh_host_ed25519_key ; /usr/sbin/sshd

I'll follow both boxes closely to see if there are any additional issues, but for the moment everything seems in order.

Many thanks!

Marcello
São Paulo - SP - Brazil

Derelict

I think the only problem would be rebooting. If those keys are stored in config.xml and overwritten on a reboot, you might lose them. I don't see anything in the config file and I can't remember if keys are regenerated on a reinstall/restore or not. I think they might be so you might be ok.

cellobita

Tried rebooting, the SSH daemon came back up without a hitch - the issue seems solved, but I'll keep a close watch on both boxes.

kejianshi

Install service watchdog and add all your services to it (like SSH).

I'd almost recommend this to be a default setting in pfsense now.

cmb

Those are the two types of keys it generates post-upgrade (they didn't exist in earlier versions). It'll generate those in the background, which delays the start of SSH until it's completed. That happens very quickly on fast CPUs, but will take some time on something like a 500 MHz Geode. Regardless, it should always still start once that is complete.

OP's issue is it somehow failed to generate those keys.

cellobita: if you still have system logs from the reboot (Diag>Command, 'clog /var/log/system.log' to dump the full history), do you see anything in there about SSH or key generation?

@NOYB:

If you run console option 14 does it change option from one to the other? Mine would not. Always stayed on enable ssh.

That menu is a bit of a hack in that regard - "Enable SSH" means sshd is not running, "Disable SSH" means sshd is running. So it's not necessarily correct with the setting in the config, though before this thread I'd yet to hear of anyone seeing it show incorrectly.

cellobita

Sorry, they seem to go back a few hours only, and the upgrade was performed days ago - whatever the logs showed, it isn't there anymore…

kejianshi

I wonder what would happen if someone cut the power or rebooted during key generation?

cmb

@kejianshi:

I wonder what would happen if someone cut the power or rebooted during key generation?

Shouldn't matter, if the keys aren't there, they'll be generated at the next boot, and they won't be put into place until they're generated.

cellobita

I must confess that I didn't try rebooting the box(es) after the upgrade - I just assumed (wrongly?) that the generation of both keys was triggered by the upgrade, and not something that would happen again if they weren't there during the startup process.

Anyway, as mentioned before all my twenty or so pfSense installs are now on 2.2, and everything is working smoothly - my sincerest thanks to the developers, I wish every other piece of software in use on my customers was as reliable as this. Kudos!

Marcello
São Paulo - SP - Brazil