(BETA 4 & 5) Connection lost during subnet routing

jojje

I'm having an issue where the firewall cuts off connections between one subnet and another on the same LAN interface after a while.

Any advice? Is this a bug?

My network looks like this:

I connect from HostA to HostB via SSH and can use the session for a couple of seconds and then the connection is killed.
To make sure it wasn't something to do with the statefulness of the beta firewall I added a "keep alive" option to the SSH client which makes it send some data every 10 seconds to avoid that specific problem. That didn't help, the connection was still cut after a minute or so.

When I finally added a static route on HostA to the firewall guarding subnet B and then the connections ceased to drop, so it's definitely something to do with the firewall (pfSense Beta5 in the diagram).

I did 4 tests to see if there was some pattern to the drops but couldn't discern a regular pattern. The four SSH connections were dropped after different periods of time (44, 83, 123 and 78 seconds respectively).

FYI my routing on the pfSense Beta5 box looks like this:

Destination Gateway Flags Refs Use Netif Expire
default a.a.a.a UGS 0 192 re0
b.b.b.b yy:yy:yy:yy:yy:yy UHS 0 23 re0
c.c.c.0/22 link#1 U 0 1017 re0
c.c.c.222 link#1 UHS 0 0 lo0
127.0.0.1 link#5 UH 0 281 lo0
172.16.0.0/24 192.168.0.100 UGS 0 231 re1
192.168.0.0/24 link#2 U 0 1641 re1
192.168.0.1 link#2 UHS 0 0 lo0
d.d.d.d zz:zz:zz:zz:zz:zz UHS 0 19 re0

re1 is my LAN interface and re0 is connected to the internet.
My two subnets are 172.16.0.0/24 and 192.168.0.0/24.
HostA has IP 192.168.0.2 and HostB 172.16.0.10 and is routed to via another pfSense (v1.2.3) firewall at 192.168.0.100

jojje

Update: problem solved (mine at least, though may still be a bug)

I found a checkbox "Bypass firewall rules for traffic on the same interface" which took care of it for my particular use case.
However I'd be very interesting to know what causes the firewall to cut the connections when this checkbox is unticked and why…

If I didn't have two separate firewalls this issue would still cause problems since ticking the box means you can't control traffic between different subnets hooked to the same NIC (interface) which I imagine is a not so uncommon configuration. E.g. where an organization has a VMWare ESX server with the promiscuous state for virtualized NICs disabled and want to hook those guests as well as the office up to the same pfSense firewall (to reduce management cost of having two separate firewalls). In such a situation there would definitely be a need for controlling traffic between the two subnets / networks (office and virtualized server farm).

cmb

You can't statefully filter asymmetrically routed traffic, that's normal.

jojje

Thanks for the reply cmb, albeit brief as it was.

Would you mind elaborating a bit on what you see as asymmetric about the depicted route?

As far as I can see, there should be no asymmetry in the routes since both outgoing and incoming traffic should travel over the same interface (re1) for the connections. As far as my limited networking knowledge goes, my understanding of the problem area described as asymmetric routing (*) is that it manifests when inbound traffic arrives on another interface than the one used for outbound, something which is (or should not be) the case here.

http://www.linuxjournal.com/article/7291

cmb

Traffic from host A to host B hits the 2.0 box, then the 1.2.3 box. Traffic in the opposite direction won't hit the 2.0 box at all. Since the 2.0 box only sees half the connection, it looks like spoofed traffic.

jojje

Ah, thank you for that clarification !
I forgot to consider that fact (that since 1.2.3 sits on the same net as the originating host, it sees no need to consult an external router but dispatches the reply traffic itself immediately to the originating host)

EDIT: In this case it might have been smarter to have a DHCP setup since the only point for me in having the 2.0 router involved is that it eliminates the need to configure an extra route on each server on the LAN. With DHCP I could just push the route dynamically from the DHCP-server (assuming pfSense has that capability, to both push routes in the 2.0 version and apply them on the WAN side via DHCP in v1.2.3).
Currently I'm wasting bandwidh funneling data via the 2.0-firewall, something which could be avoided with a static route on each machine.