Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs
-
Network:
pfSense -> Switch -> ComputerA 192.168.7.2 (VLAN 7) -> ComputerB 192.168.8.2 (VLAN 8) -> ComputerC 192.168.7.3 and 192.168.8.3 (VLAN 7 and 8)I would like to connect from ComputerA to ComputerB on VLAN 8 and to ComputerC on VLAN 8.
I have created the following firewall rule in VLAN 7:
Protocol: IPv4 * Source: 192.168.7.2 Port: * Destination: RFC1918 (All private networks) Port: * Gateway: *Trying to connect from ComputerA to ComputerB works because the rule is working. But connecting from ComputerA to ComputerC only works for 30 seconds and disconnects any SSH session with the error:
client_loop: send disconnect: Connection resetSome blocking of the SSH connections with various TCP flags are shown in the firewall log:
Action: Pass Interface: VLAN 7 Source: 192.168.7.2 Destination: 192.168.8.3:22 Protocol: TCP:SAction: Block Interface: VLAN 7 Source: 192.168.7.2 Destination: 192.168.8.3:22 Protocol: TCP:PA^ This one repeats 5 more times
Action: Block Interface: VLAN 7 Source: 192.168.7.2 Destination: 192.168.8.3:22 Protocol: TCP:RAIt seems like the issue is because ComputerC is connected to both VLAN 7 and 8, so a "shorter path" is found (VLAN 7 to VLAN 7) and the router attempts to renegotiate the connection to switch to that (TCP:PA and TCP:RA) but fails.
If I disable the VLAN 7 NIC on ComputerC then the connection sticks with VLAN 8 and doesn't disconnect.
It seems similar to the asymmetric routing issue but there is only one gateway involved (pfSense) and there are no static routes configured.
Is there a way to keep the VLAN 7 NIC enabled on ComputerC but still keep a stable connection from ComputerA to ComputerC on VLAN 8?
-
@User6buinf43 said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
Network:
pfSense -> Switch -> ComputerA 192.168.7.2 (VLAN 7)
-> ComputerB 192.168.8.2 (VLAN 8)
-> ComputerC 192.168.7.3 and 192.168.8.3 (VLAN 7 and 8)I would like to connect from ComputerA to ComputerB on VLAN 8 and to ComputerC on VLAN 8.
Why???
Computer C, which has an IP in VLAN 7 as well, will see the source IP of A, which is within VLAN 7, and will hence send the respond packets out on the VLAN 7 NIC, no matter on which NIC the gateway is defined.
Thus the respond packet will go directly to computer A and is not passing pfSense in the expected direction. Hence pfSense is missing the packets and will close the connection consequently.You can only access the VLAN 8 IP of C from A with doing masquerading (S-NAT) on pfSense if you really want this for whatever reason.
-
@User6buinf43 why would computer c need a leg in both networks? And your issue is not like asymmetric routing issue, it is a asymmetrical routing issue.
-
@viragomann said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
Why???
VLAN 7 is a VLAN with unrestricted access to the internet. VLAN 8 is the management VLAN with no internet access. ComputerA should be able to access any of the machines listening on VLAN 8. Ideally, ComputerA would also be connected to VLAN 8, and it was, but unfortunately the NIC on the motherboard has died and no PCIe lanes remain to add another NIC so ComputerA is stuck with only one NIC which has to be connected to VLAN 7 to get internet access.
@viragomann said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
You can only access the VLAN 8 IP of C from A with doing masquerading (S-NAT) on pfSense
Thank you, I will look into how to do this.
@johnpoz said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
@User6buinf43 why would computer c need a leg in both networks? And your issue is not like asymmetric routing issue, it is a asymmetrical routing issue.
I like to keep a management network (VLAN 8) separate from other networks to be used just for managing servers with SSH. It's useful to know that the issue is definitely an asymmetrical routing one.
-
@User6buinf43 said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
You can only access the VLAN 8 IP of C from A with doing masquerading (S-NAT) on pfSenseThank you, I will look into how to do this.
Firewall > NAT > Outbound
Enable the hybrid mode.
Add a rule:
Interface: VLAN8
source: 192.168.7.2
destination: 192.168.8.3 (or an alias for multiple devices if needed)
translation: VLAN8 address -
@User6buinf43 said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
I like to keep a management network (VLAN 8) separate from other networks to be used just for managing servers with SSH
huh? So you plan on putting all of your devices in multiple vlans? This defeats the whole purpose of a firewall..
Multihoming devices is almost always going to lead to asymmetrical traffic flow. Unless you are very careful and don't cross connections..
So for example I isolated 2.5ge network between my pc and nas.. This is used for transfer of files.. But there is never a scenario where the source of traffic to the nas from pc would be different network..
When you create a connection from something in your 8 vlan to a vlan 7 IP on a box that also has an IP in van 8 this is going to create asymmetrical flow - because he says oh a vlan 8 IP is talking to me, I will just answer with my vlan 8 IP..
-
@viragomann said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
Enable the hybrid mode.
Add a rule:
Interface: VLAN8
source: 192.168.7.2
destination: 192.168.8.3 (or an alias for multiple devices if needed)
translation: VLAN8 addressI've tried adding this Outbound NAT rule, choosing 192.168.8.88 as the NAT translation address, but the SSH initial connection times out. It seems partially working in that the address that reaches ComputerC is 192.168.8.88, and the iptables firewall on ComputerC is allowing the connection, but its sshd doesn't log any requests with the highest log level and doesn't respond.
When running Wireshark on ComputerC I can see that 192.168.8.88 attempts to connect to 192.168.8.3 once with the SYN flag followed by four TCP Retransmissions.
I tried changing "NAT Reflection mode for port forwards" from "disabled" to "Pure NAT" and enabling "Enable automatic outbound NAT for Reflection" but it didn't help.
The pfSense Firewall logs only mention the traffic being allowed to Pass with a SYN flag. When trying the connection, that States page shows two results for a 192.168.8.3 search:
Interface: VLAN 7 Protocol: tcp Source (Original Source) -> Destination (Original Destination): 192.168.7.2:37545 -> 192.168.8.3:22 State: CLOSED:SYN_SENT Packets: 2 / 0 Bytes: 104 B / 0 BInterface: VLAN 8 Protocol: tcp Source (Original Source) -> Destination (Original Destination): 192.168.8.88:49493 (192.168.7.2:37545) -> 192.168.8.3:22 State: SYN_SENT:CLOSED Packets: 2 / 0 Bytes: 104 B / 0 B -
@johnpoz said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
huh? So you plan on putting all of your devices in multiple vlans? This defeats the whole purpose of a firewall..
Most devices will only be on a single VLAN (VLAN 7), but some servers are connected to a second VLAN (VLAN 8) to be used for management only (SSH, iDRAC, backups, etc.). The servers have multiple NICs, one for each VLAN. This way, a server can be communicating at full link speed on both VLANs simultaneously.
@johnpoz said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
Multihoming devices is almost always going to lead to asymmetrical traffic flow. Unless you are very careful and don't cross connections..
So for example I isolated 2.5ge network between my pc and nas.. This is used for transfer of files.. But there is never a scenario where the source of traffic to the nas from pc would be different network..
Yes, it seems so. But I would like to learn how to be careful and not cross those connections.
@johnpoz said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
When you create a connection from something in your 8 vlan to a vlan 7 IP on a box that also has an IP in van 8 this is going to create asymmetrical flow - because he says oh a vlan 8 IP is talking to me, I will just answer with my vlan 8 IP..
Yes, this is occurring, but I'd like to find a way to route this.
-
@User6buinf43 said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
I tried changing "NAT Reflection mode for port forwards" from "disabled" to "Pure NAT"
Why would you think you need to create a port forward? This is a simple nat.. You do not need to port forward anything.
-
@johnpoz said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
@User6buinf43 said in Losing inter-VLAN connection after 30 seconds with one gateway and multiple NICs:
I tried changing "NAT Reflection mode for port forwards" from "disabled" to "Pure NAT"
Why would you think you need to create a port forward? This is a simple nat.. You do not need to port forward anything.
I wasn't clear on how these options fit together so I was just trying anything that seems like it could affect this.
-
I've also tried disabling the Outbound NAT rule and creating a 1:1 NAT rule:
Interface: VLAN 8 Address Family: IPv4 External subnet IP: 192.168.8.88 Internal IP: 192.168.7.2 Destination: 192.168.8.3But the result is the same as with the Outbound NAT rule, the connection enters ComputerC as 192.168.8.88, but disappears from there.
-
If you sniff on this device and see traffic coming from pfsense IP address on this network, and this device does not send an answer back - that is not on pfsense, that is on the device..
Here.. I created an outbound nat to my linux box running on my 192.168.2 network (wlan). from my lan network 192.168.9
You can see when I ssh from my 192.168.9.100 box to 192.168.2.13, it is natted to my pfsense IP on the 192.168.2 network 192.168.2.253
Here is the wireshark of the above packet capture
you can see the syn, and the box sending syn,ack back to 192.168.2.253 (pfsense ip on this wlan network as I call it).. And then starting the ssh conversation figuring out what cipher to use, etc.
But if you see pfsense send the syn, to the correct IP and mac.. And you don't get a syn,ack back - that is on the box your sshing too..
edit: I know remove the nat, and when I talk to that 2.13 device from my 9.100 device, you can see its answer to 192.168.9.100 is actually sent to pfsense mac of its 192.168.2.253 IP.. ie its gateway to get off the 2 network..
-
Thank you @johnpoz and @viragomann! I changed the Outbound NAT rule Translation Address from 192.168.8.88 to VLAN 8 address and this worked! ComputerC sees the connection as coming from the pfSense IP 192.168.8.1. I didn't understand that the address had to be the address of pfSense, not some arbitrarily chosen address.
-
@User6buinf43
You can use any free IP for masquerading in fact, but you have to assign it to the respective pfSense interface. Otherwise ARP will not work for it.I advised you to select VLAN 8 address before, however. There is no plausible reason to use any other.
