Cannot SSH over OpenVPN anymore

sotirone

Hello, I just found an issue that I haven't encountered before and I can't find anything related in the logs.

I have two pfSense firewalls connected through OpenVPN site to site. One was running 2.5.2, the other one is running 2.6.0.

At some unknown point, ssh stopped working from the 2.5.2 LAN to the 2.6.0 LAN and vice versa. I have tried 3 Arch Linux machines on the 2.5.2 side and 1 Arch and 1 Ubuntu machine on the 2.6.0 side and all failed. SSH from the 2.5.2 machine itself to the 2.6.0 machine itself fails but works the other way around. I also tried ssh with JuiceSSH on my Android smartphone but that goes through successfully for some reason!

The 2.5.2 machine is running suricata. I tried disabling suricata and clearing the blocked hosts but it did not help.

The only way I can ssh through the tunnel is by adding the

-c aes256-ctr

option which I randomly found online here: http://www.held.org.il/blog/2011/05/the-myterious-case-of-broken-ssh-client-connection-reset-by-peer/

I also tried updating the 2.5.2 box to 2.6.0 but that changed nothing as well.

If I try without the option above, ssh just fails with: "Connection closed by 10.0.0.10 port 22" . I am posting the -vvv output of the failed command below.

$ ssh root@10.0.0.10 -vvv
OpenSSH_9.0p1, OpenSSL 1.1.1o  3 May 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 10.0.0.10 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 10.0.0.10 [10.0.0.10] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /home/.ssh/id_rsa type 0
debug1: identity file /home/.ssh/id_rsa-cert type -1
debug1: identity file /home/.ssh/id_ecdsa type -1
debug1: identity file /home/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/.ssh/id_ed25519 type 3
debug1: identity file /home/.ssh/id_ed25519-cert type -1
debug1: identity file /home/.ssh/id_ed25519_sk type -1
debug1: identity file /home/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/.ssh/id_xmss type -1
debug1: identity file /home/.ssh/id_xmss-cert type -1
debug1: identity file /home/.ssh/id_dsa type -1
debug1: identity file /home/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.0
debug1: compat_banner: match: OpenSSH_9.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 10.0.0.10:22 as 'root'
debug3: record_hostkey: found key type ED25519 in file /home/.ssh/known_hosts:129
debug3: load_hostkeys_file: loaded 1 keys from 10.0.0.10
debug1: load_hostkeys: fopen /home/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 10.0.0.10 port 22

Maybe related to: https://forum.netgate.com/topic/170784/unable-to-ssh-ftp-to-the-server ?

Edit: I can confirm this started today at somepoint probably in the morning. I have a ZFS automated backup service that runs at 03:00am and that was successful. I tried running the service now manually and it failed with Connection closed.

Edit2: OK, I seem to have solved it. I have set tun-mtu 1500; to the client (the previous 2.5.2 box) and now it is working. I think the server was setting mtu at 1558. I don't know what changed today, maybe some ISP setting but ssh is working now. I also don't know if I have to set any other settings, please tell me if you think I need to.