Weird timeout issue in pfSense I please need some help figuring out

[amrogers3]

@johnpoz
I am learning something, this is cool.

It works no problem on the LAN interface. The GUEST interface is where I am having problems.

[amrogers3]

@johnpoz

I cleared the states and reconnected. Here are my results.

I connected to GUEST and here are the states:

I reset states and connected to LAN and here are my results, GUEST showed up in this state table even though I reset the states.

[johnpoz]

@amrogers3 like I said you have some device with legs in multiple networks - you are going to have problems with asymmetrical flow..

If you have a client that has a leg in 2 networks, A and B if you will and you want to talk to someting B, then use your B network to talk to it, etc..

So half the traffic doesn't flow over a different path.

Those syn_sent:closed means hey opened a state when saw the syn, but never saw any return traffic..

[amrogers3]

@johnpoz

I read the guide on troubleshooting asymmetric routing.

The app works fine on the LAN side. I don't understand why this is an issue on the GUEST side and not the LAN side. I have a static route side for GUEST. I can just run DMSS on the LAN side but I would like to learn what is going on and how to fix it.

This is my network diagram

This is my static route. I dont even have a route set up for the LAN to the cameras but the LAN can access GUEST (but GUEST cannot access LAN side)

[stephenw10]

Ah so 'GUEST' here is effectively the pfSense WAN side? And this fails when you try to connect to cameras from the GUEST subnet?

If so that traffic never goes through pfSense. What is the gateway at 192.168.1.250? Do the cameras also share that gateway device?

[amrogers3]

@stephenw10

Thanks for the questions. WAN is its own interface. Sorry I didn't specify the WAN connection on my diagram. GUEST traffic has to go through pfSense. The cameras have to be isolated from the internet.

I have three interfaces, WAN, GUEST, LAN

192.168.1.250 is my layer 3 switch. I have an IP route set up on the switch.

ip route 0.0.0.0/0 192.168.1.1

The cameras do not share that gateway. The cameras are on 192.168.3.x

The NVR is

[stephenw10]

Ok but 192.168.3.1 is also the layer 3 switch?

OK the problem here is that other hosts in the 192.168.1.X subnet (GUEST) do not have a route to 192.168.3.X so they send their traffic to the default gateway which is pfSense.

pfSense does have a route to 192.168.3.X but it's in the same subnet as the hosts so it sends back a ICMP redirect that those hosts should use the gateway dircetly. And that works but only for 30s.

It's an asymmetric traffic issue as first suspected.

The correct solution here would be to have separate transport subnet for traffic between the switch and pfSense without any other hosts on it. Then have GUEST hosts use the switch as their default gateway.

But that might be tricky to implement given the physical layout of your devices.

You can add a firewall rule on GUEST with advanced options set so that it allows the traffic when the redirect expires as a workaround:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html#manual-fix

Steve

[amrogers3]

@stephenw10
Thank you Steve, that is very helpful.
I don't fully understand everything you said. Would it be too much trouble to pop what you are explaining into a diagram. I can much better understand if I can visualize it.

My apologies, I am not a network guy. I learned everything I did with a lot of research and trial and error. Took a long time for me to figure everything out.

I think what you are suggesting would require a separate NIC?

[johnpoz]

@amrogers3 why are you running your cameras on a different ip space?

Just run your cameras on on whatever you guest network is.. 192.168.1.x, if your going to do downstream router, an L3 switch? than as Steve mentions you should have a transit network.. No you don't need another nic.. Just create a vlan transit network..

Here is how you do L3 router downstream

This transit could be specific physical network or could just be another vlan.

So in that example that 172.28.1/24 would be your camera network, and the 172.28.2/24 would be your guest network.

[amrogers3]

@johnpoz
Thank you John. I'll have to do some research and figure out how to implement this.