Issue to establish SSH connection between two different network interfaces

fabiolanza · Jan 19, 2025, 1:53 PM

Hi everyone, I am having issues to establish SSH connection between two different VLANs (each associated with a dedicated interface) on my home network. I believe to have configured everything correctly, but it does not work. I have added screenshots of most of my pfSense configuration, including the packet capture of both the SSH client and server. Any ideas on what's the configuration issue?

One potential area to clarify is the Firewall State Policy in the Advanced Options. I have tried with both Interface Bound States and Floating States, but it did not work either way. However, in a configuration like mine, which would be the most adequate state option here?

Thanks!

Client SSH Packet Capture

Server SSH Packet Capture

pfSense VLANs

pfSense Interface Assignments

pfSense Interface Definition

pfsense Firewall Rules

pfSense Outbound NAT

stephenw10 · Jan 19, 2025, 3:35 PM

Can you ping between them?

Can you ping between them with large packet sizes?

Note in the pcap that the first large packet the server side sends never reaches the client. Could be an MTU problem.

fabiolanza · Jan 19, 2025, 4:53 PM

@stephenw10 Hi, I can ping between them. Ping with default size has no packet loss. Ping with a packet size of 1508 bytes had zero packet loss either, and large packet size (e.g. 30500 bytes) generated ~11% packet loss in either side of the ping. Can this be a lead to the root cause?

fabiolanza · Jan 19, 2025, 6:14 PM

@fabiolanza

To provide more information, when initiating the SSH connection these are states

And this is the error that I obtain on the SSH client

fabio@ubuntu:~$ ssh xt3rminator@linserver.lanza.lan -vvv
OpenSSH_9.6p1 Ubuntu-3ubuntu13.5, OpenSSL 3.0.13 30 Jan 2024
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/fabio/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/fabio/.ssh/known_hosts2'
debug2: resolving "linserver.lanza.lan" port 22
debug3: resolve_host: lookup linserver.lanza.lan:22
debug3: channel_clear_timeouts: clearing
debug3: ssh_connect_direct: entering
debug1: Connecting to linserver.lanza.lan [10.0.20.11] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/fabio/.ssh/id_rsa type -1
debug1: identity file /home/fabio/.ssh/id_rsa-cert type -1
debug1: identity file /home/fabio/.ssh/id_ecdsa type -1
debug1: identity file /home/fabio/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/fabio/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/fabio/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/fabio/.ssh/id_ed25519 type -1
debug1: identity file /home/fabio/.ssh/id_ed25519-cert type -1
debug1: identity file /home/fabio/.ssh/id_ed25519_sk type -1
debug1: identity file /home/fabio/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/fabio/.ssh/id_xmss type -1
debug1: identity file /home/fabio/.ssh/id_xmss-cert type -1
debug1: identity file /home/fabio/.ssh/id_dsa type -1
debug1: identity file /home/fabio/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.6p1 Ubuntu-3ubuntu13.5
debug1: compat_banner: match: OpenSSH_9.6p1 Ubuntu-3ubuntu13.5 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to linserver.lanza.lan:22 as 'xt3rminator'
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: no algorithms matched; accept original
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-s,kex-strict-s-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: kex_choose_conf: will use strict KEX ordering
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
ssh_dispatch_run_fatal: Connection to 10.0.20.11 port 22: Broken pipe

I also tried to make the SSH connection with the following parameters to rule-out the issue with KEX algorithm pointed by the error message, but the result was the same

ssh -oKexAlgorithms=+ecdh-sha2-nistp256 xt3rminator@linserver.lanza.lan

Finally, I changed the MTU of all network interfaces to 1200 (pfSense, SSH clients and server hosts), but that did not fix it either.

Gblenn · Jan 19, 2025, 6:56 PM

@fabiolanza Perhaps clean up the rules and start with having a "default" allow "VLAN" to any rule. There is already a default deny rule, that you can't see. So with just an empty list of rules, you can do nothing from a VLAN.

A simple standard setup could be something like this for an IoTVLAN from which you might only allow access to internet, and nothing local.

Since rules 1 and 2 block any access to LAN and GuestVLAN, the only thing left is the internet in this case.

If I wanted something on IoT_VLAN to be able to access a specific device/server on my default LAN (192.168.1.10 for example). I would add an allow rule at the top, with Source - IOT_VLAN Subnet and Destination 192.168.1.10. I could add port 22 as well, to make it more specific. Or I could say Source (and specify an IP) so that only one specific device could access that server on LAN...

But there shouldn't be a need to do special Outboud NAT rules for any of this to work. pfsense does not masquerade or change ports for inter VLAN communication.
However, I think you have now set it up to NAT... which is the opposite of what you wanted. Since it shows that "cross over" symbol. Which is why you see masquerading happening in your pcap data.

If you want static port, you need to specify that in the rule, and it will change to this symbol. But, like I said, no need for any special NAT rules for inter VLAN communication to work, just set it to Auto.

fabiolanza · Jan 19, 2025, 7:06 PM

@Gblenn thank you so much for taking some time from your Sunday to answer my post.

I added allow all rules in both interfaces and placed them at the top of the rule set. I also tried disabling the outbound NAT which was there to enforce NO NAT, as for a while I suspected that there could be happening some weird NAT between the interfaces. However, I still obtain the same issue. SSH works when the connection is happening in the same VLAN, but then it has to traverse the other VLAN interface in the firewall, it does not work. If I check the blocked traffic, there's nothing related, as expected since the firewall rules are in place.

fabiolanza · Jan 19, 2025, 7:12 PM

@fabiolanza

One thing came to my mind... my Netgate device is the https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/io-ports.html. If I look at the specs (below), there are two considerations:

It says that port IGB0 is for WAN and IGB1 for LAN. In m case, I am using both for WAN purposes, as I have a dual WAN setup;
There are some other notes considerations on limitation related to speed negotiation.

Could these considerations have to do with the issue that I am experiencing?

Gblenn · Jan 19, 2025, 7:13 PM

@fabiolanza Ok, so if you place them in the same VLAN, it works, but when you move one of them over to another VLAN it stops working, correct? With a rule similar to my allow VLAN to any at the top, there should not be anything stopping it from working.
Did you try setting outbound NAT to Auto?

And have you checked the DHCP Server settings on the respective VLAN's. Could it be something there that is out of the ordinary (using KEA?).

Gblenn · Jan 19, 2025, 7:15 PM

@fabiolanza said in Issue to establish SSH connection between two different network interfaces:

@fabiolanza

One thing came to my mind... my Netgate device is the https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/io-ports.html. If I look at the specs (below), there are two considerations:

It says that port IGB0 is for WAN and IGB1 for LAN. In m case, I am using both for WAN purposes, as I have a dual WAN setup;

There are some other notes considerations on limitation related to speed negotiation.

Could these considerations have to do with the issue that I am experiencing?

I doubt it, but someone else may be able to verify. I think those notes are mainly about the physical connection...

stephenw10 · Jan 19, 2025, 7:32 PM

If it was either a NAT or policy routing issue I'd expect to see the connection fail completely. But here the initial TCP handshake completes correctly. There is two way traffic.

Also you would see NAT in the states that are opened and you don't.

Try pcaps in pfSense on each interface and see if the initial large packet from the server makes it to either.

fabiolanza · Jan 19, 2025, 9:25 PM

@Gblenn I did try outbound NAT to automatic, it did not work the same way... DHCP works correctly on both interfaces, I can't think how it could be the issue, but thanks!

Gblenn · Jan 20, 2025, 8:08 AM

@fabiolanza Since it works if both are on the same subnet, but not when they are in different. Perhaps you should check for some settings on the server. Like if there are limitations in /etc/ssh/sshd_config that may prevents access from different subnets etc. Look for allowed users, IP ranges and things that may limit access.
Perhaps you can find something by comparing the logs and pcap output from when they are in the same subnet where it's working...

fabiolanza · Jan 20, 2025, 9:09 AM

@Gblenn SSH is not the only service not working. I tried to iperf3 between both hosts across the interfaces and that fails too... this is indication that it's something to do with the network. I have added firewall rules for iperf3 and I see the hits in the allow rule... it is possible to buy a single support ticket from Netgate without having to pay the yearly subscription?

Gblenn · Jan 20, 2025, 10:06 AM

@fabiolanza Ok, that's interesting... so what does your network look like, pfsense and switches?
Does pinging between the hosts work?
If you have a rule as mentioned, allow all to any, adding one for iperf3 will do nothing to help.
What do your rules look like right now, at both VLAN's?

stephenw10 · Jan 20, 2025, 1:16 PM

Do you see the same failure between any two hosts on those subnets?

fabiolanza · Jan 20, 2025, 11:31 PM

@stephenw10 and @Gblenn let me add more information...

If I place my VMs on the same VLAN, the communication between them works without issues, including the SSH connection between them and other services.

When I try to connect to the VMs from a different VLAN, services like SSH and IPERF3 will not work. Looking at the pfSense firewall logs, I see that TCP/S passes as expected, but then TCP/RA will get blocked. I added a floating rule to pass TCP/RA but it did not change anything.

Please see an image of my network diagram for more details.

Thanks

stephenw10 · Jan 21, 2025, 1:00 AM

Is the layer3 switch routing between those VLANs? If is that could create asymmetric traffic in pfSense.

But run pcaps in pfSense so you can whats actually going through both the interfaces. The earlier pcap show the server sends the Key Exchange Init packet but the client never receives it. So does that arrive at pfSense at all?