SSHD failed to start
-
@netblues
So I have worked on it some more.
Permissions for the ssh folder are correct, I havedrwxr-xr-x 4 root wheel 9B Feb 16 01:04 sslI have also been monitoring the system log file and only get:
Feb 17 01:24:12 pfplus_ARK1550 check_reload_status[410]: starting sshdOn restart sshd is not running. It still will not start when I hit the button in the widget, however if I run /usr/sbin/sshd in command prompt I get ssh keys not available however sshd will start. However, on restart sshd does not auto start and I have to run /usr/sbin/sshd again.
At this point I would get the following occasionally in the log.
Also get php-fpm[671]: /sshd: Pushover API server did not return data in expected format!After attempting all these steps I completed a fresh install and restored from config and I am continuing to have the same issues.
I believe that this issue actually started for me when I performed a clean install of version 2.5.2 with ZFS as the file system. That was the point that I upgraded to 2.6 then to 22.01,
-
Do you actually see the process running before it starts?
If you run
ps -auxwwddo you see sshd spawning some other process which is stuck maybe? Ot itself spawned by something that is stuck?Do you have any sort of exotic crypto off loading that might be doing something odd?
Steve
-
After reboot or stopping and attempting to restart sshd
ps -auxwwdgives no reference to sshd. After running/usr/sbin/sshdI do seeroot 86270 0.0 0.1 19568 9128 - Ss 02:22 0:00.00 |-- /usr/sbin/sshdbut no other sub-processes.
I haven't made any modifications to the crypto settings.
-
Hmm, so it does it at anytime it's restarted even when the keys already exist?
-
That is correct.
The only way I am able to get it to start is by running the sbin command, and it gives me a missing key error but then sshd starts up and works.
I’m unable to start sshd using VGA terminal inputs, via the services widget, or automatically when pfsense restarts.
-
What's the exact error it shows?
-
Could not load host key: /etc/ssh/ssh_host_rsa_key
-
I had the same SSHD issue with clean installs of 2.6.0, as mentioned here.
Looking at your "ls" output earlier, you don't have an 'ssh_host_rsa_key' - which is what 2.6.0 seems it now looks for - so create one:
cd /etc/ssh ssh-keygen -N '' -t rsa -f ssh_host_rsa_key /usr/sbin/sshd...try a reboot after that and it should now be fixed.
-
@manicmoose
While a good suggestion, I attempted it and have the same issue after reboot. It did generate the key, but after restart sshd didn't start, and running /usr/sbin/sshd still tells me the key is missing.Also as a side note I have been having this issue since performing a clean install of 2.5.2
-
@kryptickahos What are the actual contents of your sshd config?
cat /etc/ssh/sshd_config -
# This file is automatically generated at startup KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 Port 22 Protocol 2 HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_ed25519_key Compression delayed ClientAliveInterval 30 PermitRootLogin yes # Login via Key or Password ChallengeResponseAuthentication yes PasswordAuthentication yes PubkeyAuthentication yes UseDNS no LoginGraceTime 30s VersionAddendum none AllowAgentForwarding no X11Forwarding no Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com # override default of no subsystems Subsystem sftp /usr/libexec/sftp-server -
@kryptickahos
Okay - so that looks identical to mine so should be fine.
The only difference is that I'm running 'CA' 2.6.0, not 22.01+I doubt that would matter, so it's a bit of a strange one.
Now that you've generated the 'rsa' key, does the error still say the same thing, ie.
Could not load host key: /etc/ssh/ssh_host_rsa_key ? -
@manicmoose
Since then I have reverted back to CE on 2.6.0
And yes, after generating ssh_host_rsa_key I still get the error when starting sshd. -
@kryptickahos
Well, it's going to be a strange outcome.If you've got the file that pfSense is whinging about and the perm's are all correct, then there's something probably only Netgate can answer going wrong.
If you can't get any joy here then I guess you'll have to re-install from scratch and not restore your config. Good luck...! -
I managed to get sshd working as expected. Failure for the keys to be generated correctly appears have been related to expiration of webConfigurator default certificate.
I renewed the self signed cert, disable ssh, rebooted, re-enabled ssh. Received the keys are being generated notification and then immediately keys were generated and now sshd works correctly.
-
@kryptickahos
Hmmm....thought the expired cert would have come up as a pretty obvious alert on the web console page.
Either way, it would have been nice to see this error also in the logs to give you a clue.
Nice you've sorted it, anyhow!!
-
@manicmoose
It was obvious that the cert was expired, just didn't think it would have anything to do with ssh key generation. I also incorrectly assumed it was from an old cert when I was playing with the acme plugin.It's still weird I was able to start ssh from command prompt, and it gave a the missing key error, but either way I'm glad its working now.
-
Mmm, that is weird. I can't really see how an expired cert would affect that. I'll see if I can replicate it.
Steve
-
@stephenw10
One final piece of information that may help with the actual root cause of the issue. When troubleshooting I also found an old static IP entry on LAN for the device I was using as my firewall. This was an old entry from when I was using a different firewall and had this device on the network. During the time of getting the fix working I also deleted the static IP entry, but it didn't immediately resolve the issue so I didn't think it was the solution but it might have been.I attempted further testing by restoring to the old config that had the static IP as well as the expired self signed cert (I didn't perform a clean install this time). At reboot after restoring with both issues sshd worked correctly.
So this may actually be related to the static IP entry but only manifests after a clean install and restore. If I get some time in the future I may attempt a clean install again but as of current this is the best info I have.
As an extra point the firewall I'm using is a dual Ethernet port device, with non-switched ports. I'm not sure if the WAN or LAN port was assigned the static IP, but it was one of them.
-
I have the impression that we have several issues that somehow interact here.
First, it seems that (for our systems) after an upgrade from 2.5.2 to 2.6, SSH host keys get lost. I have verified this for my installation (with a clean install) already.
Second, a re-install from USB (with config.xml recovery from disk) has the same issue. I have verified this as well: A manual backup of the config.xml before doing the reinstall has the keys; after the reinstall, they seem to be lost.
Third, although keys are in place (e.g. by running
ssh-keygen -Afrom the Diagnostics/Command Prompt), sshd fails to start (logs look ok, Status/Services indicates a red X, running
ssh admin@pfsenseagainst the pfSense host times out.
However, when running
/usr/sbin/sshdfrom the Diagnostics/Command Prompt manually, it seems to start fine (no output at all), and sshing into the machine works fine.
After rebooting a few times (usually once doesn't suffice), sshd seems to start normally.
Let me know if I can provide you with additional information (logs etc.); I am happy to help.
