Issue to establish SSH connection between two different network interfaces
-
Can you ping between them?
Can you ping between them with large packet sizes?
Note in the pcap that the first large packet the server side sends never reaches the client. Could be an MTU problem.
-
@stephenw10 Hi, I can ping between them. Ping with default size has no packet loss. Ping with a packet size of 1508 bytes had zero packet loss either, and large packet size (e.g. 30500 bytes) generated ~11% packet loss in either side of the ping. Can this be a lead to the root cause?
-
To provide more information, when initiating the SSH connection these are states
And this is the error that I obtain on the SSH client
fabio@ubuntu:~$ ssh xt3rminator@linserver.lanza.lan -vvv OpenSSH_9.6p1 Ubuntu-3ubuntu13.5, OpenSSL 3.0.13 30 Jan 2024 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files debug1: /etc/ssh/ssh_config line 21: Applying options for * debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/fabio/.ssh/known_hosts' debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/fabio/.ssh/known_hosts2' debug2: resolving "linserver.lanza.lan" port 22 debug3: resolve_host: lookup linserver.lanza.lan:22 debug3: channel_clear_timeouts: clearing debug3: ssh_connect_direct: entering debug1: Connecting to linserver.lanza.lan [10.0.20.11] port 22. debug3: set_sock_tos: set socket 3 IP_TOS 0x10 debug1: Connection established. debug1: identity file /home/fabio/.ssh/id_rsa type -1 debug1: identity file /home/fabio/.ssh/id_rsa-cert type -1 debug1: identity file /home/fabio/.ssh/id_ecdsa type -1 debug1: identity file /home/fabio/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/fabio/.ssh/id_ecdsa_sk type -1 debug1: identity file /home/fabio/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /home/fabio/.ssh/id_ed25519 type -1 debug1: identity file /home/fabio/.ssh/id_ed25519-cert type -1 debug1: identity file /home/fabio/.ssh/id_ed25519_sk type -1 debug1: identity file /home/fabio/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /home/fabio/.ssh/id_xmss type -1 debug1: identity file /home/fabio/.ssh/id_xmss-cert type -1 debug1: identity file /home/fabio/.ssh/id_dsa type -1 debug1: identity file /home/fabio/.ssh/id_dsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.5 debug1: Remote protocol version 2.0, remote software version OpenSSH_9.6p1 Ubuntu-3ubuntu13.5 debug1: compat_banner: match: OpenSSH_9.6p1 Ubuntu-3ubuntu13.5 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to linserver.lanza.lan:22 as 'xt3rminator' debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug3: order_hostkeyalgs: no algorithms matched; accept original debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com,zlib debug2: compression stoc: none,zlib@openssh.com,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-s,kex-strict-s-v00@openssh.com debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com debug2: compression stoc: none,zlib@openssh.com debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug3: kex_choose_conf: will use strict KEX ordering debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com debug1: kex: host key algorithm: ssh-ed25519 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY ssh_dispatch_run_fatal: Connection to 10.0.20.11 port 22: Broken pipeI also tried to make the SSH connection with the following parameters to rule-out the issue with KEX algorithm pointed by the error message, but the result was the same
ssh -oKexAlgorithms=+ecdh-sha2-nistp256 xt3rminator@linserver.lanza.lanFinally, I changed the MTU of all network interfaces to 1200 (pfSense, SSH clients and server hosts), but that did not fix it either.
-
@fabiolanza Perhaps clean up the rules and start with having a "default" allow "VLAN" to any rule. There is already a default deny rule, that you can't see. So with just an empty list of rules, you can do nothing from a VLAN.
A simple standard setup could be something like this for an IoTVLAN from which you might only allow access to internet, and nothing local.
Since rules 1 and 2 block any access to LAN and GuestVLAN, the only thing left is the internet in this case.
If I wanted something on IoT_VLAN to be able to access a specific device/server on my default LAN (192.168.1.10 for example). I would add an allow rule at the top, with Source - IOT_VLAN Subnet and Destination 192.168.1.10. I could add port 22 as well, to make it more specific. Or I could say Source (and specify an IP) so that only one specific device could access that server on LAN...
But there shouldn't be a need to do special Outboud NAT rules for any of this to work. pfsense does not masquerade or change ports for inter VLAN communication.
However, I think you have now set it up to NAT... which is the opposite of what you wanted. Since it shows that "cross over" symbol. Which is why you see masquerading happening in your pcap data.
If you want static port, you need to specify that in the rule, and it will change to this symbol. But, like I said, no need for any special NAT rules for inter VLAN communication to work, just set it to Auto.
-
@Gblenn thank you so much for taking some time from your Sunday to answer my post.
I added allow all rules in both interfaces and placed them at the top of the rule set. I also tried disabling the outbound NAT which was there to enforce NO NAT, as for a while I suspected that there could be happening some weird NAT between the interfaces. However, I still obtain the same issue. SSH works when the connection is happening in the same VLAN, but then it has to traverse the other VLAN interface in the firewall, it does not work. If I check the blocked traffic, there's nothing related, as expected since the firewall rules are in place.
-
One thing came to my mind... my Netgate device is the https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/io-ports.html. If I look at the specs (below), there are two considerations:
- It says that port IGB0 is for WAN and IGB1 for LAN. In m case, I am using both for WAN purposes, as I have a dual WAN setup;
- There are some other notes considerations on limitation related to speed negotiation.
Could these considerations have to do with the issue that I am experiencing?
-
@fabiolanza Ok, so if you place them in the same VLAN, it works, but when you move one of them over to another VLAN it stops working, correct? With a rule similar to my allow VLAN to any at the top, there should not be anything stopping it from working.
Did you try setting outbound NAT to Auto?And have you checked the DHCP Server settings on the respective VLAN's. Could it be something there that is out of the ordinary (using KEA?).
-
@fabiolanza said in Issue to establish SSH connection between two different network interfaces:
One thing came to my mind... my Netgate device is the https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/io-ports.html. If I look at the specs (below), there are two considerations:
- It says that port IGB0 is for WAN and IGB1 for LAN. In m case, I am using both for WAN purposes, as I have a dual WAN setup;
- There are some other notes considerations on limitation related to speed negotiation.
Could these considerations have to do with the issue that I am experiencing?
I doubt it, but someone else may be able to verify. I think those notes are mainly about the physical connection...
-
If it was either a NAT or policy routing issue I'd expect to see the connection fail completely. But here the initial TCP handshake completes correctly. There is two way traffic.
Also you would see NAT in the states that are opened and you don't.
Try pcaps in pfSense on each interface and see if the initial large packet from the server makes it to either.
-
@Gblenn I did try outbound NAT to automatic, it did not work the same way... DHCP works correctly on both interfaces, I can't think how it could be the issue, but thanks!
-
@fabiolanza Since it works if both are on the same subnet, but not when they are in different. Perhaps you should check for some settings on the server. Like if there are limitations in /etc/ssh/sshd_config that may prevents access from different subnets etc. Look for allowed users, IP ranges and things that may limit access.
Perhaps you can find something by comparing the logs and pcap output from when they are in the same subnet where it's working... -
@Gblenn SSH is not the only service not working. I tried to iperf3 between both hosts across the interfaces and that fails too... this is indication that it's something to do with the network. I have added firewall rules for iperf3 and I see the hits in the allow rule... it is possible to buy a single support ticket from Netgate without having to pay the yearly subscription?
-
@fabiolanza Ok, that's interesting... so what does your network look like, pfsense and switches?
Does pinging between the hosts work?
If you have a rule as mentioned, allow all to any, adding one for iperf3 will do nothing to help.
What do your rules look like right now, at both VLAN's? -
Do you see the same failure between any two hosts on those subnets?
-
@stephenw10 and @Gblenn let me add more information...
If I place my VMs on the same VLAN, the communication between them works without issues, including the SSH connection between them and other services.
When I try to connect to the VMs from a different VLAN, services like SSH and IPERF3 will not work. Looking at the pfSense firewall logs, I see that TCP/S passes as expected, but then TCP/RA will get blocked. I added a floating rule to pass TCP/RA but it did not change anything.
Please see an image of my network diagram for more details.
Thanks
-
Is the layer3 switch routing between those VLANs? If is that could create asymmetric traffic in pfSense.
But run pcaps in pfSense so you can whats actually going through both the interfaces. The earlier pcap show the server sends the Key Exchange Init packet but the client never receives it. So does that arrive at pfSense at all?
