Weird timeout issue in pfSense I please need some help figuring out

[amrogers3]

I can't figure out how to troubleshoot this.

I have an app (DMSS used to view camera feeds) that uses port 37777 to connect to the cameras. I have two NICs A and B. I opened up 37777 in both and able to connect to the cameras using either NIC but the video times out in NIC B after 30 seconds. Both rules in NIC A and B for 37777 are the same. I dont get an error in DMSS, the video just stops.

I don't see anything in the firewall rules that would make it time out after 30 seconds. Also, nothing stands out in the packet capture.

Any helpful suggestions on how to troubleshoot this issue or what would cause pfSense to do this?

[johnpoz]

@amrogers3 30 second time out points to asymmetrical traffic flow.. Since this is the default timeout for a tcp session to fully open.. Pfsense sees the syn and open the state.. But if the rerturn traffic is not flowing back through pfsense the state would time out and and close.. Now the client would have to send a syn again.. Rinse and repeat..

You mention multiple nics, I take it this is in your client - so yeah that also screams asymmetrical flow..

If you have a client that has legs in multiple networks.. You really need to understand your traffic flow patterns or with a stateful firewall your going to run into problems depending..

[amrogers3]

This is my home system. I have a NIC for lan devices like NAS and such. The other NIC is for TVs, IoT, etc.

The device isn't in multiple networks. I basically disconnected from one NIC and joined the other to do the testing.

I have the cameras on a separate VLAN and have a static route to reach the cameras from either NIC.

I set up both NICs the same so this seems unusual behavior. What would be the best way to troubleshoot this or force pfSense to keep the connection open.

[johnpoz]

@amrogers3 said in Weird timeout issue in pfSense I please need some help figuring out:

troubleshoot this or force pfSense to keep the connection open.

Have the return traffic flow through pfsense.. You wouldn't want to force open a asymmetrical state.

Look at your state table for your connection.. Do you see it established:established?

That is my pc talking to one of my video cameras.. Here is the stream of video connection

here it is streaming video - again notice established:established... If your not seeing that while your watching video, then its not open fully.. And yeah will prob time out in 30 seconds.. What do you show for the state?

[amrogers3]

@johnpoz
I am learning something, this is cool.

It works no problem on the LAN interface. The GUEST interface is where I am having problems.

[amrogers3]

@johnpoz

I cleared the states and reconnected. Here are my results.

I connected to GUEST and here are the states:

I reset states and connected to LAN and here are my results, GUEST showed up in this state table even though I reset the states.

[johnpoz]

@amrogers3 like I said you have some device with legs in multiple networks - you are going to have problems with asymmetrical flow..

If you have a client that has a leg in 2 networks, A and B if you will and you want to talk to someting B, then use your B network to talk to it, etc..

So half the traffic doesn't flow over a different path.

Those syn_sent:closed means hey opened a state when saw the syn, but never saw any return traffic..

[amrogers3]

@johnpoz

I read the guide on troubleshooting asymmetric routing.

The app works fine on the LAN side. I don't understand why this is an issue on the GUEST side and not the LAN side. I have a static route side for GUEST. I can just run DMSS on the LAN side but I would like to learn what is going on and how to fix it.

This is my network diagram

This is my static route. I dont even have a route set up for the LAN to the cameras but the LAN can access GUEST (but GUEST cannot access LAN side)

[stephenw10]

Ah so 'GUEST' here is effectively the pfSense WAN side? And this fails when you try to connect to cameras from the GUEST subnet?

If so that traffic never goes through pfSense. What is the gateway at 192.168.1.250? Do the cameras also share that gateway device?

[amrogers3]

@stephenw10

Thanks for the questions. WAN is its own interface. Sorry I didn't specify the WAN connection on my diagram. GUEST traffic has to go through pfSense. The cameras have to be isolated from the internet.

I have three interfaces, WAN, GUEST, LAN

192.168.1.250 is my layer 3 switch. I have an IP route set up on the switch.

ip route 0.0.0.0/0 192.168.1.1

The cameras do not share that gateway. The cameras are on 192.168.3.x

The NVR is

[stephenw10]

Ok but 192.168.3.1 is also the layer 3 switch?

OK the problem here is that other hosts in the 192.168.1.X subnet (GUEST) do not have a route to 192.168.3.X so they send their traffic to the default gateway which is pfSense.

pfSense does have a route to 192.168.3.X but it's in the same subnet as the hosts so it sends back a ICMP redirect that those hosts should use the gateway dircetly. And that works but only for 30s.

It's an asymmetric traffic issue as first suspected.

The correct solution here would be to have separate transport subnet for traffic between the switch and pfSense without any other hosts on it. Then have GUEST hosts use the switch as their default gateway.

But that might be tricky to implement given the physical layout of your devices.

You can add a firewall rule on GUEST with advanced options set so that it allows the traffic when the redirect expires as a workaround:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html#manual-fix

Steve

[amrogers3]

@stephenw10
Thank you Steve, that is very helpful.
I don't fully understand everything you said. Would it be too much trouble to pop what you are explaining into a diagram. I can much better understand if I can visualize it.

My apologies, I am not a network guy. I learned everything I did with a lot of research and trial and error. Took a long time for me to figure everything out.

I think what you are suggesting would require a separate NIC?

[johnpoz]

@amrogers3 why are you running your cameras on a different ip space?

Just run your cameras on on whatever you guest network is.. 192.168.1.x, if your going to do downstream router, an L3 switch? than as Steve mentions you should have a transit network.. No you don't need another nic.. Just create a vlan transit network..

Here is how you do L3 router downstream

This transit could be specific physical network or could just be another vlan.

So in that example that 172.28.1/24 would be your camera network, and the 172.28.2/24 would be your guest network.

[amrogers3]

@johnpoz
Thank you John. I'll have to do some research and figure out how to implement this.