TCP timeouts and drops through static routes

jkramp

Hi,

My first post here, and I'm a bit insecure under which category this should be placed, if this is completely wrong please feel free to move it.

I have issue, that I think is caused by static routing.

On my LAN (10.1.0.0/24), pfSense (10.1.0.10) is acting as DHCP server and Firewall with ISP1 connected directly to pfSense WAN marked as default gateway, then I have a Cisco Catalyst 3560G routing some network to ISP2 which is 10.1.0.1, that I have added under System > Routing > Gateways
Under System > Routing > Routes I have added the desired networks to the 10.1.0.1 gateway and traffic flows just fine.

Here comes the trouble, when I'm using eg. CIFS/SMB, RDP & HTTP over the static routes, I get weird timeouts and drops - then it renegotiates and sometimes it just keep doing that every 30 sec. most of the times it renegotiates 2 times and become stable. The RDP issue is way worse to Windows Server 2012 R2 than it's to Windows Server 2008 R2 and I don't recall ever having a drop for Windows Server 2003 R2.

I can reproduce this in my test pfSense environment as well, but if I switch in my OpenWRT with the same static routes I never have these drops.

I have googled a lot and found that Checking "Bypass firewall rules for traffic on the same interface
This option only applies if you have defined one or more static routes. If it is enabled, traffic that enters and leaves through the same interface will not be checked by the firewall. This may be desirable in some situations where multiple subnets are connected to the same interface." it worked out quite a lot for like a month, then the drops and time-out returned.

Then I unchecked "The monitoring process will flush states for a gateway that goes down if this box is not checked. Check this box to disable this behavior." and it was stable for like 24 hours.

I know the easy thought is unstable GW's, but they are not, ping never drops nor does my Avaya IP (UDP) phones running through the static routes.

Does any of the above make any sense to some you?

Help is very much appreciated and thank you in advance.

BR. Jørgen

phil.davis

This looks like it has a good bit of banter about your asymmetric routing issue: https://forum.pfsense.org/index.php?topic=30717.0

The returning packets from Cisco get delivered directly to the client on the LAN. pfSense never sees them. It has created states for the traffic, but after seeing nothing back for a while it times out the state. Then you get a glitch.

Put a rule high up on LAN to match traffic going across the static route to the Cisco and select Advanced, State type, sloppy state. That will stop pfSense from fussing about it.

UDP and ICMP work OK because they are not stateful like TCP is. So pfSense can't fuss as much about seeing 2-way flow for those.

Personally, I would get rid of asymmetric routing if possible. Put a separate interface with a little subnet on pfSense that connects to the Cisco. Route stuff across that, take the Cisco off the local LAN. Then the Cisco will have to route back through pfSense. pfSense sees all and is happy - no way for Cisco to bypass pfSense.

jkramp

Hi Phil,

Thank very much for your reply!

I'll ask the ISP2 to add a transition NW to the Cisco and add it as a VLAN to my LAN, that's the right way to do it then, right?

/Jørgen

phil.davis

It's a pity the Cisco is not under your direct control.
To do it, the Cisco will need to understand that it has no local interface to your LAN subnet, otherwise it will keep trying to deliver packets directly to that.
What they need to do to the Cisco for you is:

Change the IP/subnet of the current Cisco LAN away from 10.1.0.1/24 - lets say they make it 10.12.34.1/24
(You make an interface in pfSense with interface 10.12.34.2/24 and gateway 10.12.34.1)
Add a route in the Cisco to tell it that 10.1.0.0/24 is reached through 10.12.34.2 (the pfSense)
Do whatever they need to do to pass traffic to/from 10.1.0.0/24

Unplug the cable from the Cisco to LAN, connect it from Cisco to pfSense new interface.
Put your static route/s to point to gateway 10.12.34.1
Put any rules you want on the new interface to allow incoming connections (if you need/want to allow connections to be started from out there behind the Cisco to LAN devices)

I think that about covers it.

The main thing I wanted to point out is that "transition" will be a bit difficult, because the Cisco is going to have to be told a new route to your LAN, and thus lose its current direct LAN interface - I don't think the 2 can co-exist in the Cisco config nicely.

jkramp

It's only for period of 3 weeks, then our Check Point cluster will move location to where the pfSense is spinning today.

I have made the fw rule with sloppy state now and it seems to be working, at least right now - so that might be the solution for the next 3 weeks, thanks a bunch! :)