SSH connections timing out to other networks behind pfsense

bdf0506

My network is setup like this:


Modem -> PfSense -> Switch -> HostA
                         |--> Cisco ASA -> Host B

My PfSense sits in the 192.168.200.0/24 network, as does the Switch and Host A.
The Cisco ASA (which firewalls off my lab from everything else) has its outside interface on 192.168.200.0/24, and the inside interface is on 192.168.250.0/24. Host B sits on 192.168.250.0/24.
My hardware for PfSense only has two NICs, so one NIC connects to the Modem, the other to the switch. So all my 192.168.x.x traffic is on the LAN side of pfSesne.

In my old setup, with Tomato, I had no issues with SSH (so I have no reason to believe that the ASA is causing these issues). But with this new setup, I find that SSH sessions going from Host A to Host B are timing out after about 10 minutes of being idle, and I am not sure as to why.

I've checked out a few different threads, and I've attempted a couple of things, like setting Firewall Optimization Options to Conservative, clear invalid DF bits instead of dropping the packets, disabled Firewall Scrub… but nothing seems to help. I'm guessing that if I turn off the packet filtering entirely that this will probably solve my issue, but then that will defeat the purpose of even having the pfSense in the first place.

If I am on a host that is on 192.168.200.0/24 and connect to Host A, the session will stay alive all day. It's only until I go from a host on 192.168.200.0/24 to Host B where I have the issues.

Any thoughts or things I may be missing?

Derelict

The default inactivity timeout of an established TCP session in pfSense is 24 hours so if you are seeing 10 minutes something else is causing it.

Check the state in Diagnostics > States. After a disconnection see if the state still exists there.

I can assure you pfSense does not kill such states unless it is told to do so. I have ssh sessions established for weeks.

USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
derelict pts/1 192.168.223.6 Sat14 7.00s 0.40s 0.01s w

bdf0506

It looks like something isn't allowing the handshake to fully happen. But, the SSH connection does stay established for a minute or two, likely until the packet gets killed. Perhaps I need an allow rule somewhere that lets the traffic really establish. Any thoughts on the best way to fix this?


States
Interface	Protocol	Source (Original Source) -> Destination (Original Destination)	State	Packets	Bytes	
LAN	tcp	192.168.200.101:54124 -> 192.168.250.97:22	CLOSED:SYN_SENT	33 / 0	5 KiB / 0 B	
LAN	tcp	192.168.200.101:54124 -> 192.168.250.97:22	SYN_SENT:CLOSED	33 / 0	5 KiB / 0 B

Derelict

It looks like you have created an asymmetric routing situation. Don't do that. You should not have hosts and routers on the same network. The link to the ASA should be a transit network consisting of nothing but router interfaces.

This is based on the limited information at-hand.

bdf0506

I am still struggling with this. I'm not convinced there is asymetric routing going on.

If I attempt the ssh session from the pfsense itself, you can see that everything is fine, and we show established.
But if I attempt the ssh session from a host connected to the pfsense, I have issues.

The host in my case - 192.168.200.101, has a gateway which is the pfSense itself, 192.168.200.1

States
Interface	Protocol	Source (Original Source) -> Destination (Original Destination)	State	Packets	Bytes	
LAN	tcp	192.168.200.1:63005 -> 192.168.250.97:22	ESTABLISHED:ESTABLISHED	24 / 22	3 KiB / 4 KiB	
LAN	tcp	192.168.200.101:63930 -> 192.168.250.97:22	CLOSED:SYN_SENT	19 / 0	3 KiB / 0 B	
LAN	tcp	192.168.200.101:63930 -> 192.168.250.97:22	SYN_SENT:CLOSED	19 / 0	3 KiB / 0 B

Routing table on pfsense (192.168.200.10 is the outside iface of the ASA):

Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
192.168.200.0/24    link#2             U           re1
192.168.250.0/24    192.168.200.10      UGS         re1

A traceroute from my Windows client further proves that we are taking the same path that the pfSense is taking.

tracert 192.168.50.97

Tracing route to Server01.bdf.local [192.168.250.97]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  pfSense.home [192.168.200.1]
  2     2 ms     1 ms     1 ms  CiscoASA.home [192.168.200.10]
  3     3 ms     2 ms     2 ms  Server01.bdf.local [192.168.250.97]

Trace complete.

I can try to create a new network that simply links the pfsense to the ASA, though I am not convinced that will fix this issue.

Derelict

Host A either has the ASA as its default gateway or pfSense. Which is it? Does it have a route in its own routing table for the network behind the ASA? Is the ASA natting?

You will likely need to provide a better diagram. See my sig for one with the information one might provide in a diagram they expect someone else to be able to understand quickly.

I can assure you that there is zero problem with ssh sessions through pfsense in a default configuration.

bdf0506

Thanks. I did repost this in the NAT section since it's likely not a firewall section any longer (couldn't figure out a way to move the topic).

Host A has the pfsense as the DG, and no special static routes are on Host A itself. It is depending on the pfSense to route it to the ASA. I am going to try to give the LAN interface of pfsense a VLAN and a different transport network for the pfsense <-> ASA link per your suggestion last week to see if that does fix anything.

Derelict

Host A has the pfsense as the DG, and no special static routes are on Host A itself. It is depending on the pfSense to route it to the ASA.

Right. That is considered asymmetric routing. pfSense has to make a special case for routing traffic back out the same interface it arrived on which a router does not normally do. It might also be that you can initially communicate from Host A to Host B because Host A receives an ICMP redirect from pfSense telling it to communicate directly with Host B via the ASA's address. That expires on Host A and the session goes off the rails. In fact that makes the most sense.

You can try checking Static route filtering - Bypass firewall rules for traffic on the same interface in System > Advanced, Firewall and NAT.

Even if you enable Static route filtering the ICMP redirect will still be sent and might still be honored unless you also set this sysctl net.inet.ip.redirect: 0 in System > Advanced, System Tunables.

That's why your asymmetric routing is bad.

I would redesign it properly instead of relying on a bunch of special checkboxes and sysctls to get around the poor design.

See the image. Host A would be the Management host, The Cisco ASA would be the Layer 3 Switch, and Host B would be behind it.

http://www.cymru.com/gillsr/documents/icmp-redirects-are-bad.htm

pfSense-Layer-3-Switch-WT.png_thumb

bdf0506

Thanks for the pointers here. I see what you mean with it going out the same interface and how it had issues with the design. I am guessing that Tomato just simply did not care, and had an option behind the scenes or by default where it didn't care about traffic going in and out the same interface. Either way, as you mentioned earlier, it's better to have a proper network design, instead of trying to hack together something.

I went ahead and created a VLAN/additional network to handle the link from the pfsense to the ASA, and now it's all good.


States
Interface	Protocol	Source (Original Source) -> Destination (Original Destination)	State	Packets	Bytes	
LAN	tcp	192.168.200.101:51565 -> 192.168.250.97:22	ESTABLISHED:ESTABLISHED	21 / 21	4 KiB / 5 KiB	
PFSENSE_ASA	tcp	192.168.200.101:51565 -> 192.168.250.97:22	ESTABLISHED:ESTABLISHED	21 / 21	4 KiB / 5 KiB

Thanks!!