Weird interaction between pfSense and MikroTik router

jrandombob · Apr 9, 2019, 6:51 AM

Hi All,

Got a bit of a weird one, hoping somebody might be able to provide some suggestions.

I've got a pfSense box at my network edge, and a MikroTik router sitting in front of all my wireless kit.

On pfSense I've got some static routes defined to push traffic bound for the networks on the other side of the MikroTik to its LAN-facing interface.

The weird issue I'm running into is that if I connect to a device on a wireless subnet (or the wireless management subnet) from my LAN I'm able to connect fine, but regardless of what (if anything) is going on on the connection, about 50 seconds later my connection gets dropped.

If I define a static route on the machine I'm connecting from to push traffic bound for the wireless subnet directly to the LAN-facing interface on the MikroTik the connection stays up and happy. So it seems that the MikroTik side of things is working fine but something's weird with pfSense.

When the connection drops I see a flurry of TCP Retransmissions from my workstation to the target device and a few seconds later the connection gets reset.

There's a LAN to * allow rule in the firewall so it shouldn't be anything in that area.

I've not seen this behaviour on previous iterations of my setup which used different routers in place of the MikroTik so it appears to be some weird interaction between the two.

Any suggestions would be appreciated.

Thanks,

-J

JKnott · Apr 9, 2019, 10:40 AM

Do you actually get traffic with DHCP? An address? It should make no difference whether you have a static or DHCP address and once you have it via DHCP, it's valid for the entire lease time. It seems more likely DHCP is failing to get an address and the connection times out.

stephenw10 · Apr 9, 2019, 11:23 AM

You are seeing the result of asymmetric routing:
https://docs.netgate.com/pfsense/en/latest/firewall/troubleshooting-blocked-log-entries-due-to-asymmetric-routing.html

You can try the 'Bypass firewall rules for traffic on the same interface' option suggested there. It would be better to remove the asymmetry though if you can, connect the MikroTik to pfSense via a different interface. You may be able to use a VLAN on the LAN port for example.

Steve

jrandombob · Apr 9, 2019, 9:37 PM

@JKnott I'm not sure you're responding to the post you think you're responding to, this issue has nothing to do with DHCP, but thanks anyway.

jrandombob · Apr 9, 2019, 11:36 AM

@stephenw10 Thanks for the suggestions, I'll have a go and see what happens.

But based on that doco I'd say you're probably right, connections being dropped from the table after 30 seconds would make sense given the symptoms I'm seeing (though the timing is a bit off).

stephenw10 · Apr 9, 2019, 1:03 PM

The actual timing depends on a number of things there. I'd say it's almost certain that's what you're hitting though.

Steve

jrandombob · Apr 9, 2019, 9:01 PM

@stephenw10 Yeah, I'd expect as much, I'll poke at it over the next few days and see how it goes. Thanks

JKnott · Apr 9, 2019, 9:37 PM

@jrandombob said in Weird interaction between pfSense and MikroTik router:

@JKnott I'm not sure you're responding to the post you think you're responding to, this issue has nothing to do with DHCP, but thanks anyway.

You said it works with static IP, but not DHCP? Once a device has an address via DHCP, there is no difference than having a static IP, for the duration of the lease. If it fails after 50 s with DHCP, then that would indicate a problem with DHCP. You mentioned the wireless devices are on the other side of the MikroTik router from the LAN? Is there a DHCP server there? If not, you'll get a failed connection after several seconds.

jrandombob · Apr 10, 2019, 12:30 PM

Added a Floating rule and a LAN rule with sloppy state set per the doco, works like a charm.

At some point I'll rearrange my network to hang the wireless off a different interface on pfSense, but for the moment this does what I need it to.

Thanks @stephenw10 for the pointer.

NetVicious · Feb 27, 2020, 8:05 AM

Hi @jrandombob, can you explain a bit more the solution you found?
I don't understand what it's the floating rule and the sloppy state.
Regards,

jrandombob · Feb 27, 2020, 10:35 AM

@NetVicious Under Firewall->Rules you'll find a "Floating" tab along with all your other interfaces. You can define rules here which aren't tied to a specific interface (not 100% correct explanation but good enough for these purposes).

As for "sloppy state", when you're configuring the firewall rules, there's a "State type" option under "Advanced Options" one of the options there is "Sloppy", basically setting that makes the state matching for established connections related to the rule less strict.

The "Manual Fix" section on this page provides more or less step-by-step instructions;
https://docs.netgate.com/pfsense/en/latest/firewall/troubleshooting-blocked-log-entries-due-to-asymmetric-routing.html

NetVicious · Feb 27, 2020, 11:20 AM

Thanks for the explanation, it's not my exact scenario but will help others.