Basic Routing Question
-
I'm pretty sure this is going to be a simple answer but I'm looking for some validation on my setup to make sure I'm doing this correctly. I've been seeing some issues with dropped packets and it makes me think I'm doing something incorrectly like asym routing.
I have a pfSense system with 4 NICs. I have:
-
eth0 (WAN) - PPPoE from my fibre modem
-
eth1 (LAN) - Static IP of 192.168.100.254
-
eth2 (MGT) - Static IP of 192.168.110.254
-
eth3 - unused at the moment
eth1 and eth2 are both plugged into a managed switch. Each interface is in a set of ports which is bound to a VLAN for segregation. eth1 is in VLAN100, eth2 is in VLAN110. Since these are seperate NICs and into a managed switch with VLANs configured, I have not used VLAN tagging in pfSense as my switch is configured to tag. Everything seems to be working on the surface but I'm getting some odd issues like packets dropping if I ssh from a server in the LAN to a server in the MGT network.
It might be because I don't have any static routes or gateways defined (except for WAN which is set to default gateway). I thought that if I set a server in LAN with a gateway address of the eth1 IP and a server in the MGT with a gateway address of the eth2 IP things everything would be fine. Is this correct? Should I be using static routes and gateways? I assumed that since both the interfaces were defined in pfSense that I wouldn't need to do this but perhaps I am wrong?
Any advice would be greatly appreciated.
Cheers.
Tom.
-
-
You shouldn't need static routes between pfSense interfaces. The LAN interface has a default Allow All firewall rule so it can access all interfaces, and default behaviour is to allow return traffic so nothing should be blocked. Can we see soem examples of these dropped packets? I'm wondering if it's just out of state packets being dropped.
https://doc.pfsense.org/index.php/Why_do_my_logs_show_%22blocked%22_for_traffic_from_a_legitimate_connection
-
Ah OK - thanks for clarifying the static route question. I amended my LAN to any rule so that only a single IP can access the MGT network and I have the rule set to any TCP/UDP from LAN server IP to MGT network. I'm sure this rule is OK as I can access the services in MGT with this rule in place (IPMI, remote KVM and the like).
I've just checked the firewall logs again and I'm noticing something I hadn't before, I seem to now be getting blocked on outbound from MGT to the LAN. Strange, before, it was blocking TCP:PA and TCP:RA from LAN to MGT.
Attached is a screenshot of the log. 2213 is the port I am using for SSH connections.
I have the state killing and skip gateway rules options enabled and have also tried with sloppy states on the TCP/UDP rule to no avail.
Regards.
Tom.
-
That looks all to be out of state traffic. Nothing to worry about. Any time you see Push ACK or Reset ACK being blocked on trusted interfaces, it's almost always this.
-
Is there any indication of trouble other than the logs?
-
The only thing is that when I'm connected to a remote server using either SFTP or SCP using WinSCP, it drops and reconnects every few minutes making it almost impossible to transfer files.
-
I have ssh sessions that stay connected for days/weeks. SSH has a keepalive function that keeps firewall states active.
I would undo all the clicky-clicky in pfSense you have done and work on the applications themselves.
-
OK - I will have a look into the application config to see if their is something configurable. I seem to remember there being something like 'connection optimization buffer' or some such in WinSCP - I'll check it out.
It is weird though as I have had several pfSense servers over the last few years, each servicing a similar network topology and it's only since the latest build (new build for 2.2.4) that I've had this issue.
Regards.
t.
-
I have noticed no difference in 2.2.4.
-
So given my config above, you don't think this is the asym. routing issue I've been seen mentioned on the forums? I only ask as the problem seems similar to others.
-
I also run ssh and scp sessions that can last for days. No issues at all with dropped connections. I didn't need to twiddle any app-specific settings either. Check your firewall logs around the time when a connection drops to see if there's anything going on.
-
You have to be dealing with two routers to have an asymmetric routing problem. As long as pfSense is the default gateway on both segments you should be fine.
I haven't had to twiddle any settings in ssh/sshd either. At least not for several years.
-
From the looks of that, the 110.4 host is probably dual homed on both networks, leaving the host itself creating an asymmetric path.
-
Ahh, that may well be it. Yes the 110.4 host is dual homed.
It is another FreeBSD box (NAS4Free). It has eth0 connected to VLAN110 and eth1 connected to VLAN100. It is configured to use 192.168.110.254 as it's default gateway. I suspect that is wrong then?
-
Am I right in assuming that I don't need to have any trunked ports on my switch as I have a interface from pfSense directly connected into each VLAN segment?
I'm reading up about VLAN routing and most of the guides are for pfSense with single LAN interfaces.
-
OK - so it definitely looks like asymmetrical routing is the issue here.
If I disable the second NIC attached to VLAN100 on the server I am trying to SSH to, it removes the route on that server for 192.168.100.0/24 and I no longer have an issue with SSH. Works perfectly. I can see the packets going from 192.168.100.50 (my PC) > 192.168.110.254 > 192.168.110.4 and then back along the same path as the primary NIC in the server is on the 110 VLAN with 192.168.110.254 as the default gateway.
Am I doing something fundamentally wrong here. I thought the packet would come back the way it entered but I guess not. Is there any way I can achieve this without losing the ability to keep both NICs on the server I want to SSH to? Ideally I want to keep them for dedicated jail usage on the seperate subnets.
-
In general you can use the other interfaces for same-subnet traffic but anything outside a subnet that your NAS thinks is "connected" is going to go to the default gateway unless there is a specific route for it in the NAS.
Your jails might be able to have a different default gateway from the main NAS routing table. I'm not sure.
-
Am I doing something fundamentally wrong here. I thought the packet would come back the way it entered but I guess not.
No, return traffic follows the system routing table, which means it sends it out the directly-connected network in that circumstance.
If you dual home a system like that, don't connect cross-subnet to it. Strictly connect to its IP on the same subnet as the source machine, or the IP of the interface where the default gateway resides if it's sourced from a network where the destination system isn't directly connected. That'll ensure no issues along those lines.
