pfsense-ce 2.7.4 SSH server: how to config ClientAliveCountMax and ClientAliveInterval

oldunixguy

I have many linux servers running OpenSSH and configure (/etc/ssh/sshd_config) the server as:
TCPKeepAlive no
ClientAliveInterval 60
ClientAliveCountMax 5

And on the client SSH used with these systems I have TCP Keep Alive disabled and Server Alive Interval disabled. I rely on the server to detect the comm failure and the server to initiate the null packet request sent to the client.

On pfsense I see there is a file /etc/ssh/sshd_config BUT it reports in the top that this is overwritten at boot. It does not say where the originating config items are kept such that one can make changes there.

How do I set these 3 pfsense server options?

thanks,
oldunixguy

stephenw10

It's created by /etc/sshd: https://github.com/pfsense/pfsense/blob/RELENG_2_7_2/src/etc/sshd#L72

If you create the file /etc/sshd_extra and put your lines there they should be read in and included:
https://github.com/pfsense/pfsense/blob/RELENG_2_7_2/src/etc/sshd#L123

oldunixguy

@stephenw10
thanks for the reply and proposal. I did place these settings in the /etc/sshd_extras and they do appear in /etc/ssh/sshd_config after boot.
However, I noticed the runtime config file now has 2 ClientAliveInterval. An "original" with value 30 and now mine at the end with value 60.
Does this sshd use the "last one set"?

I haven't run any tests yet to confirm if it works as intended. I will report back once tested.

thanks
oldunixguy

Gertjan

@oldunixguy

Normally, yes.
If a second identical settings like "ClientAliveCountMax" (with the same or another value) couldn't be taken in account, there would be an error and ssh refused to start.
Its very common to see a global config file for a process, and at the end this global config file it "sources" = includes other 'user' config files so every user can override certain behaviors, without having them to edit the global config file.

oldunixguy

@stephenw10
Well, I have done testing and found interesting and unexpected results!
If I use these:
TCPKeepAlive no
ClientAliveInterval 60
ClientAliveCountMax 5

I get a 4-packet interaction with no data every 60 seconds.
No. Time Source Destination Protocol Length Info
1 19:13:17.934396101 192.168.0.1 192.168.0.100 TCP 146 EtherNet-IP-1(2222) → 50324 [PSH, ACK] Seq=1 Ack=1 Win=514 Len=80 TSval=1106488073 TSecr=23442467
2 19:13:17.934467400 192.168.0.100 192.168.0.1 TCP 66 50324 → EtherNet-IP-1(2222) [ACK] Seq=1 Ack=81 Win=433 Len=0 TSval=23457485 TSecr=1106488073
3 19:13:17.934854830 192.168.0.100 192.168.0.1 TCP 114 50324 → EtherNet-IP-1(2222) [PSH, ACK] Seq=1 Ack=81 Win=433 Len=48 TSval=23457485 TSecr=1106488073
4 19:13:17.934987922 192.168.0.1 192.168.0.100 TCP 66 EtherNet-IP-1(2222) → 50324 [ACK] Seq=81 Ack=49 Win=514 Len=0 TSval=1106488073 TSecr=23457485
5 19:14:17.990412665 192.168.0.1 192.168.0.100 TCP 146 EtherNet-IP-1(2222) → 50324 [PSH, ACK] Seq=81 Ack=49 Win=514 Len=80 TSval=1106548129 TSecr=23457485
6 19:14:17.990730251 192.168.0.100 192.168.0.1 TCP 114 50324 → EtherNet-IP-1(2222) [PSH, ACK] Seq=49 Ack=161 Win=433 Len=48 TSval=23472499 TSecr=1106548129
7 19:14:17.990885455 192.168.0.1 192.168.0.100 TCP 66 EtherNet-IP-1(2222) → 50324 [ACK] Seq=161 Ack=97 Win=514 Len=0 TSval=1106548129 TSecr=23472499

But if I use these:
TCPKeepAlive no
ClientAliveInterval 577
ClientAliveCountMax 5

I get a 3-packet interaction with no data EVERY 60 SECONDS TOO!
No. Time Source Destination Protocol Length Info
1 18:44:06.611970133 192.168.0.1 192.168.0.100 TCP 146 EtherNet-IP-1(2222) → 33162 [PSH, ACK] Seq=1 Ack=1 Win=514 Len=80 TSval=1333636851 TSecr=23004634
2 18:44:06.612549986 192.168.0.100 192.168.0.1 TCP 114 33162 → EtherNet-IP-1(2222) [PSH, ACK] Seq=1 Ack=81 Win=368 Len=48 TSval=23019655 TSecr=1333636851
3 18:44:06.612703368 192.168.0.1 192.168.0.100 TCP 66 EtherNet-IP-1(2222) → 33162 [ACK] Seq=81 Ack=49 Win=514 Len=0 TSval=1333636851 TSecr=23019655
4 18:45:06.718264889 192.168.0.1 192.168.0.100 TCP 146 EtherNet-IP-1(2222) → 33162 [PSH, ACK] Seq=81 Ack=49 Win=514 Len=80 TSval=1333696958 TSecr=23019655
5 18:45:06.718555272 192.168.0.100 192.168.0.1 TCP 114 33162 → EtherNet-IP-1(2222) [PSH, ACK] Seq=49 Ack=161 Win=368 Len=48 TSval=23034681 TSecr=1333696958
6 18:45:06.718715827 192.168.0.1 192.168.0.100 TCP 66 EtherNet-IP-1(2222) → 33162 [ACK] Seq=161 Ack=97 Win=514 Len=0 TSval=1333696958 TSecr=23034681

With yet another combo:
TCPKeepAlive yes
ClientAliveInterval 577
ClientAliveCountMax 5

I get another surprise. I get a 4-packet interaction with no data STILL every 60 seconds.
No. Time Source Destination Protocol Length Info
1 19:24:35.388744615 192.168.0.1 192.168.0.100 TCP 146 EtherNet-IP-1(2222) → 55827 [PSH, ACK] Seq=1 Ack=1 Win=514 Len=80 TSval=3386117535 TSecr=23611825
2 19:24:35.388796613 192.168.0.100 192.168.0.1 TCP 66 55827 → EtherNet-IP-1(2222) [ACK] Seq=1 Ack=81 Win=329 Len=0 TSval=23626849 TSecr=3386117535
3 19:24:35.389019805 192.168.0.100 192.168.0.1 TCP 114 55827 → EtherNet-IP-1(2222) [PSH, ACK] Seq=1 Ack=81 Win=329 Len=48 TSval=23626849 TSecr=3386117535
4 19:24:35.389069586 192.168.0.1 192.168.0.100 TCP 66 EtherNet-IP-1(2222) → 55827 [ACK] Seq=81 Ack=49 Win=514 Len=0 TSval=3386117535 TSecr=23626849
5 19:25:35.440817650 192.168.0.1 192.168.0.100 TCP 146 EtherNet-IP-1(2222) → 55827 [PSH, ACK] Seq=81 Ack=49 Win=514 Len=80 TSval=3386177587 TSecr=23626849
6 19:25:35.441252260 192.168.0.100 192.168.0.1 TCP 114 55827 → EtherNet-IP-1(2222) [PSH, ACK] Seq=49 Ack=161 Win=329 Len=48 TSval=23641862 TSecr=3386177587
7 19:25:35.441428552 192.168.0.1 192.168.0.100 TCP 66 EtherNet-IP-1(2222) → 55827 [ACK] Seq=161 Ack=97 Win=514 Len=0 TSval=3386177588 TSecr=23641862

I conclude from this that changing TCPKeepAlive does not affect the time interval and does not change whether I get 4 packets or 3 packets.

Setting a large ClientAliveInterval such that it should not emit in 60 seconds does NOT work regardless of the TCPKeepAlive setting. It still emits at 60 seconds.

And what is this variation with 3 or 4 packets?

It looks like these are not working at all.
regards
oldunixguy

stephenw10

Are you sure it's not the other side sending keep alives? I assume 2222 is the server port?

oldunixguy

@stephenw10
This design has been used for years with our clients and and openssh servers.
The client is NOT configured for generating TCPKeepAlives nor ServerAliveInterval or CountMax. Instead the server is initiating the keep alives. In this test case the server is 192.168.0.1. In all the test cases you can see in the traces that the server initiates the keep alive probe. the client end is 192.168.0.100. The traces reflect the proper sequences but NOT the proper time intervals. The 3-packet versus 4-packet matter is yet to be studied by me and as an operational matter not that interesting. The time intervals are the important matter.
thanks
oldunixguy

stephenw10

Hmm, I would probably try removing the duplicate ClientAliveInterval value then. Just comment out the line:
https://github.com/pfsense/pfsense/blob/RELENG_2_7_2/src/etc/sshd#L82

Or just set it to 60 there directly as a test.

oldunixguy

Well, I'm back and the problem still exists. I just tested pfsense ce 2.8 and it does NOT work.

I enter the folloeing in /etc/sshd_extra
TCPKeepAlive no
ClientAliveInterval 60
ClientAliveCountMax 5

reboot

then examine the /etc/ssh/sshd_config

In it there is NO TCPKeepAlive at all, even commented.
There IS a ClientAliveInterval 30 BUT this is not correct value.
There is NO ClientAliveCountMax at all, even commented.

With other sshd implementations if there is not an entry for a value, say TCPKeepAlive, then an internal DEFAULT is used.

From previous report it seems the default TCPKeepAlive is "active".

So, the procedure to "alter" or "add" sshd config values here is BROKEN.

Could someone produce a bug report so this can get fixed?
thanks
oldunixguy

Gertjan

@oldunixguy

I created the file : /etc/sshd_extra and added :
#test line 1
#test line 2
TCPKeepAlive no
ClientAliveInterval 60
ClientAliveCountMax 5

Then I unchecked :

and saved, (the SSH connection stopped, that's normal) then checked it again, and Saved.

I entered SSH again, and :

[25.07-RC][root@pfSense.bhf.tld]/root: cat /etc/ssh/sshd_config
# This file is automatically generated at startup
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
....... (snipped) .....
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
# override default of no subsystems
Subsystem       sftp    /usr/libexec/sftp-server
#test line 1
#test line 2
TCPKeepAlive no
ClientAliveInterval 60
ClientAliveCountMax 5

so the 'extra' lines were added.

Btw : I do presume that 25.07 (pfSense Plus) and 2.8.0, pfSense CE, behave the same way ...

The script to include the "sshd_extra info" is still there in 'master' : https://github.com/pfsense/pfsense/blob/master/src/etc/sshd

patient0

@oldunixguy disclaimer: I have no used these settings before but I was interested and did some reading and testing.

Why are the ClientAlive* settings not applied: FreeBSD Manual Pages: sshd_config states:

"Unless noted otherwise, for each keyword, the first obtained value will be used."

Since ClientAliveInterval is already set by pfSense, the later added custom setting has no effect.

One way to make it work for me is to use the Match directive to overwrite these settings. The above mentioned man pages list which directive can be overwritten with Match, the ClientAlive* are on the list.

My /etc/sshd_extra looked like:

TCPKeepAlive no
Match Address *
        ClientAliveInterval 60
        ClientAliveCountMax 5

Since TCPKeepAlive is not set by pfSense, overwriting will work (but is not allowed in Match). You can restrict Address to subnets if that is necessary. E.g.

Match Address 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

... to only apply it non-WAN addresses.

I'm not sure about the TCP flow to expect. Testing with a Debian client and connecting to pfSense CE 2.8.1-BETA and a Debian 12 server, both with the same Match statements, resulted in the same flow: first a 4 packages exchange and 3 packages after that.

Gertjan

@patient0 said in pfsense-ce 2.7.4 SSH server: how to config ClientAliveCountMax and ClientAliveInterval:

"Unless noted otherwise, for each keyword, the first obtained value will be used."

Nice catch :
Thanks - that made me remember : just adding parameters hoping that "the last one is taken in account", which is something I did presume - doesn't work.
So, this is a "sshd" issue.

Editing "https://github.com/pfsense/pfsense/blob/master/src/etc/sshd" (make a patch for it so it auto applies) after an pfSense upgrade/update) will do the job.
The match trick is also a good idea and worth testing.