SSHD failed to start
-
I've been having this issue for a while and thought I got it resolved but after my last restart apparently not.
Typically this occurs after restoring my pfsense box from a backup. I will get the sshd has not started and keys are being generated and after a full day of being up sshd final starts but I don't recieve the notification that the keys have been generated. I was able to find some instructions in an older netgate forums post here, but I have to run them every time I restart my box. Also interesting when I run the keygen commands it says the keys exist. I did just upgrade to 22.01 on a whitebox and this issue still persists.
Previously I have fixed this issue my renaming my box and keys are regenerated and stay on restart, but this time it's not working. I have also attempted to stop and start sshd from local terminal on the box.
I've searched the forums and haven't found anything else to help. Does anyone have any idea what could be wrong, or why it could take so long for sshd to start, and that I get the prompt for keygen after every restart?
-
@kryptickahos check file permissions. They shouldn be readable by group/anone
/root: ls -lh /etc/ssh
total 72
-rw-r--r-- 1 root wheel 552K Feb 7 18:56 moduli
-rw-r--r-- 1 root wheel 1.5K Feb 7 18:56 ssh_config
-rw------- 1 root wheel 419B Dec 27 19:13 ssh_host_ed25519_key
-rw------- 1 root wheel 104B Dec 27 19:13 ssh_host_ed25519_key.pub
-rw------- 1 root wheel 3.3K Dec 27 19:13 ssh_host_rsa_key
-rw------- 1 root wheel 748B Dec 27 19:13 ssh_host_rsa_key.pub
-rw-r--r-- 1 root wheel 823B Feb 15 14:03 sshd_config -
Running
ls -lh /etc/sshGives
total 72 -rw-r--r-- 1 root wheel 552K Feb 7 10:56 moduli -rw-r--r-- 1 root wheel 1.5K Feb 7 10:56 ssh_config -rw------- 1 root wheel 525B Feb 15 23:43 ssh_host_ecdsa_key -rw-r--r-- 1 root wheel 189B Feb 15 23:43 ssh_host_ecdsa_key.pub -rw------- 1 root wheel 419B Feb 15 23:44 ssh_host_ed25519_key -rw-r--r-- 1 root wheel 109B Feb 15 23:44 ssh_host_ed25519_key.pub -rw-r--r-- 1 root wheel 825B Feb 16 01:04 sshd_configThis is after I run the other commands and the keys are regenerated. This state clears after rebooting.
Before that when SSHD wasn't running I got a total of 173. -
Well, as long as your key files are private to root, its not a permissions issue.
I also believe that parent folder is ok too
It should be
drwxr-xr-x 4 root wheel 7B Feb 15 05:16 sslCheck /var/log/system.log for clues regarding ssh
Looks like it is something else, disk/os related.
Since in 22.01 disk format changes to zfs, its a valid idea to do a
fresh install from usb.
Restoring the configuration from previous installations while installing from usb/console, is also straight forward. -
@netblues
So I have worked on it some more.
Permissions for the ssh folder are correct, I havedrwxr-xr-x 4 root wheel 9B Feb 16 01:04 sslI have also been monitoring the system log file and only get:
Feb 17 01:24:12 pfplus_ARK1550 check_reload_status[410]: starting sshdOn restart sshd is not running. It still will not start when I hit the button in the widget, however if I run /usr/sbin/sshd in command prompt I get ssh keys not available however sshd will start. However, on restart sshd does not auto start and I have to run /usr/sbin/sshd again.
At this point I would get the following occasionally in the log.
Also get php-fpm[671]: /sshd: Pushover API server did not return data in expected format!After attempting all these steps I completed a fresh install and restored from config and I am continuing to have the same issues.
I believe that this issue actually started for me when I performed a clean install of version 2.5.2 with ZFS as the file system. That was the point that I upgraded to 2.6 then to 22.01,
-
Do you actually see the process running before it starts?
If you run
ps -auxwwddo you see sshd spawning some other process which is stuck maybe? Ot itself spawned by something that is stuck?Do you have any sort of exotic crypto off loading that might be doing something odd?
Steve
-
After reboot or stopping and attempting to restart sshd
ps -auxwwdgives no reference to sshd. After running/usr/sbin/sshdI do seeroot 86270 0.0 0.1 19568 9128 - Ss 02:22 0:00.00 |-- /usr/sbin/sshdbut no other sub-processes.
I haven't made any modifications to the crypto settings.
-
Hmm, so it does it at anytime it's restarted even when the keys already exist?
-
That is correct.
The only way I am able to get it to start is by running the sbin command, and it gives me a missing key error but then sshd starts up and works.
I’m unable to start sshd using VGA terminal inputs, via the services widget, or automatically when pfsense restarts.
-
What's the exact error it shows?
-
Could not load host key: /etc/ssh/ssh_host_rsa_key
-
I had the same SSHD issue with clean installs of 2.6.0, as mentioned here.
Looking at your "ls" output earlier, you don't have an 'ssh_host_rsa_key' - which is what 2.6.0 seems it now looks for - so create one:
cd /etc/ssh ssh-keygen -N '' -t rsa -f ssh_host_rsa_key /usr/sbin/sshd...try a reboot after that and it should now be fixed.
-
@manicmoose
While a good suggestion, I attempted it and have the same issue after reboot. It did generate the key, but after restart sshd didn't start, and running /usr/sbin/sshd still tells me the key is missing.Also as a side note I have been having this issue since performing a clean install of 2.5.2
-
@kryptickahos What are the actual contents of your sshd config?
cat /etc/ssh/sshd_config -
# This file is automatically generated at startup KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 Port 22 Protocol 2 HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_ed25519_key Compression delayed ClientAliveInterval 30 PermitRootLogin yes # Login via Key or Password ChallengeResponseAuthentication yes PasswordAuthentication yes PubkeyAuthentication yes UseDNS no LoginGraceTime 30s VersionAddendum none AllowAgentForwarding no X11Forwarding no Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com # override default of no subsystems Subsystem sftp /usr/libexec/sftp-server -
@kryptickahos
Okay - so that looks identical to mine so should be fine.
The only difference is that I'm running 'CA' 2.6.0, not 22.01+I doubt that would matter, so it's a bit of a strange one.
Now that you've generated the 'rsa' key, does the error still say the same thing, ie.
Could not load host key: /etc/ssh/ssh_host_rsa_key ? -
@manicmoose
Since then I have reverted back to CE on 2.6.0
And yes, after generating ssh_host_rsa_key I still get the error when starting sshd. -
@kryptickahos
Well, it's going to be a strange outcome.If you've got the file that pfSense is whinging about and the perm's are all correct, then there's something probably only Netgate can answer going wrong.
If you can't get any joy here then I guess you'll have to re-install from scratch and not restore your config. Good luck...! -
I managed to get sshd working as expected. Failure for the keys to be generated correctly appears have been related to expiration of webConfigurator default certificate.
I renewed the self signed cert, disable ssh, rebooted, re-enabled ssh. Received the keys are being generated notification and then immediately keys were generated and now sshd works correctly.
-
@kryptickahos
Hmmm....thought the expired cert would have come up as a pretty obvious alert on the web console page.
Either way, it would have been nice to see this error also in the logs to give you a clue.
Nice you've sorted it, anyhow!!
