SSHD failed to start

netblues

Well, as long as your key files are private to root, its not a permissions issue.
I also believe that parent folder is ok too
It should be
drwxr-xr-x 4 root wheel 7B Feb 15 05:16 ssl

Check /var/log/system.log for clues regarding ssh

Looks like it is something else, disk/os related.
Since in 22.01 disk format changes to zfs, its a valid idea to do a
fresh install from usb.
Restoring the configuration from previous installations while installing from usb/console, is also straight forward.

KrypticKahos

@netblues
So I have worked on it some more.
Permissions for the ssh folder are correct, I have

drwxr-xr-x  4 root  wheel        9B Feb 16 01:04 ssl

I have also been monitoring the system log file and only get:

Feb 17 01:24:12 pfplus_ARK1550 check_reload_status[410]: starting sshd

On restart sshd is not running. It still will not start when I hit the button in the widget, however if I run /usr/sbin/sshd in command prompt I get ssh keys not available however sshd will start. However, on restart sshd does not auto start and I have to run /usr/sbin/sshd again.

At this point I would get the following occasionally in the log.

Also get php-fpm[671]: /sshd: Pushover API server did not return data in expected format!

After attempting all these steps I completed a fresh install and restored from config and I am continuing to have the same issues.

I believe that this issue actually started for me when I performed a clean install of version 2.5.2 with ZFS as the file system. That was the point that I upgraded to 2.6 then to 22.01,

stephenw10

Do you actually see the process running before it starts?

If you run ps -auxwwd do you see sshd spawning some other process which is stuck maybe? Ot itself spawned by something that is stuck?

Do you have any sort of exotic crypto off loading that might be doing something odd?

Steve

KrypticKahos

@stephenw10

After reboot or stopping and attempting to restart sshd ps -auxwwd gives no reference to sshd. After running /usr/sbin/sshd I do see

root    86270   0.0  0.1  19568   9128  -  Ss   02:22      0:00.00 |-- /usr/sbin/sshd

but no other sub-processes.

I haven't made any modifications to the crypto settings.

stephenw10

Hmm, so it does it at anytime it's restarted even when the keys already exist?

KrypticKahos

@stephenw10

That is correct.

The only way I am able to get it to start is by running the sbin command, and it gives me a missing key error but then sshd starts up and works.

I’m unable to start sshd using VGA terminal inputs, via the services widget, or automatically when pfsense restarts.

stephenw10

What's the exact error it shows?

KrypticKahos

@stephenw10

Could not load host key: /etc/ssh/ssh_host_rsa_key

manicmoose

@kryptickahos

I had the same SSHD issue with clean installs of 2.6.0, as mentioned here.

Looking at your "ls" output earlier, you don't have an 'ssh_host_rsa_key' - which is what 2.6.0 seems it now looks for - so create one:

cd /etc/ssh
ssh-keygen -N '' -t rsa -f ssh_host_rsa_key
/usr/sbin/sshd

...try a reboot after that and it should now be fixed.

KrypticKahos

@manicmoose
While a good suggestion, I attempted it and have the same issue after reboot. It did generate the key, but after restart sshd didn't start, and running /usr/sbin/sshd still tells me the key is missing.

Also as a side note I have been having this issue since performing a clean install of 2.5.2

manicmoose

@kryptickahos What are the actual contents of your sshd config?

cat /etc/ssh/sshd_config

KrypticKahos

@manicmoose

# This file is automatically generated at startup
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
Compression delayed
ClientAliveInterval 30
PermitRootLogin yes
# Login via Key or Password
ChallengeResponseAuthentication yes
PasswordAuthentication yes
PubkeyAuthentication yes
UseDNS no
LoginGraceTime 30s
VersionAddendum none
AllowAgentForwarding no
X11Forwarding no
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
# override default of no subsystems
Subsystem	sftp	/usr/libexec/sftp-server

manicmoose

@kryptickahos
Okay - so that looks identical to mine so should be fine.
The only difference is that I'm running 'CA' 2.6.0, not 22.01+

I doubt that would matter, so it's a bit of a strange one.
Now that you've generated the 'rsa' key, does the error still say the same thing, ie.
Could not load host key: /etc/ssh/ssh_host_rsa_key ?

KrypticKahos

@manicmoose
Since then I have reverted back to CE on 2.6.0
And yes, after generating ssh_host_rsa_key I still get the error when starting sshd.

manicmoose

@kryptickahos
Well, it's going to be a strange outcome.

If you've got the file that pfSense is whinging about and the perm's are all correct, then there's something probably only Netgate can answer going wrong.
If you can't get any joy here then I guess you'll have to re-install from scratch and not restore your config. Good luck...!

KrypticKahos

I managed to get sshd working as expected. Failure for the keys to be generated correctly appears have been related to expiration of webConfigurator default certificate.

I renewed the self signed cert, disable ssh, rebooted, re-enabled ssh. Received the keys are being generated notification and then immediately keys were generated and now sshd works correctly.

manicmoose

@kryptickahos
Hmmm....thought the expired cert would have come up as a pretty obvious alert on the web console page.
Either way, it would have been nice to see this error also in the logs to give you a clue.
Nice you've sorted it, anyhow!!

KrypticKahos

@manicmoose
It was obvious that the cert was expired, just didn't think it would have anything to do with ssh key generation. I also incorrectly assumed it was from an old cert when I was playing with the acme plugin.

It's still weird I was able to start ssh from command prompt, and it gave a the missing key error, but either way I'm glad its working now.

stephenw10

Mmm, that is weird. I can't really see how an expired cert would affect that. I'll see if I can replicate it.

Steve

KrypticKahos

@stephenw10
One final piece of information that may help with the actual root cause of the issue. When troubleshooting I also found an old static IP entry on LAN for the device I was using as my firewall. This was an old entry from when I was using a different firewall and had this device on the network. During the time of getting the fix working I also deleted the static IP entry, but it didn't immediately resolve the issue so I didn't think it was the solution but it might have been.

I attempted further testing by restoring to the old config that had the static IP as well as the expired self signed cert (I didn't perform a clean install this time). At reboot after restoring with both issues sshd worked correctly.

So this may actually be related to the static IP entry but only manifests after a clean install and restore. If I get some time in the future I may attempt a clean install again but as of current this is the best info I have.

As an extra point the firewall I'm using is a dual Ethernet port device, with non-switched ports. I'm not sure if the WAN or LAN port was assigned the static IP, but it was one of them.