How is pfSense preventing this traffic on the same subnet?
The below diagram is the relevant portion of the firewall config I'm having trouble with. The actual device is multi-wan + multi-lan, but the traffic I'm having trouble with is limited to a single subnet.
All LAN devices are connected to the switch. The switch is connected to the LAN interface on pfSense. I have the following firewall rule on the LAN interface:
TCP/IP Version: IPv4
Destination: Alias for all RFC1918 networks
The above rule is intended to block traffic between multiple LANs.
The problem I'm having is that when I enable the above rule, it has the effect of sometimes preventing traffic between client 1 and client 2, but not client 2 and client 3. How is that possible? All 3 machines are on the same subnet. Unfortunately I only tested ICMP traffic between the machines and can't risk breaking anything to do more testing until I have a better understanding of what's happening.
The actual testing I've done has me very confused… I can ping (-t) between client 1 and client 2 consistently until I try to open a specific application on client 2. As soon as that application tries to load, and communicate with client 1, traffic between the 2 machines stops and pinging times out (request timeout). However, as soon as the application fails to load and terminates, ICMP traffic between the two machines returns to normal.
Even more confusing to me, I'm able to ping between client 2 and client 3 the entire time I'm trying to launch the mentioned application and there is no interruption.
What could possibly cause the above scenario?
Things are not how you think they are. It's not possible for pfSense to block such traffic. Check your switch and your config.
Actually, it sounds like your problem is that app, not pfSense. Not that it even CAN be pfSense.
Thanks for the reply Derelict. I agree. However, enabling and disabling of that rule definitely has an impact. Perhaps I shouldn't have said pfSense is preventing the traffic, rather that my blocking rule has some kind of side effect that I don't understand.
Since you've verified I haven't gone crazy, my current best guess is that the mentioned application is using the machine's gateway (pfSense LAN interface) to check connectivity and everything breaks when it thinks the gateway is down due to being blocked by my rule.
The only part of that hypothesis that doesn't make sense to me is why ICMP would break between client 1 and client 2 when the application is trying to start. The application would have to be doing something pretty weird to have that effect, wouldn't it?
Yeah, but who knows what it's doing. If the application is stopping the network stack, then that's what it's doing. I'm not saying the rule isn't causing your problem. What I'm saying is pfSense isn't blocking the ICMP between the two hosts on the same subnet. That simply cannot be.
Yes, your application/OS is doing something weird.
Thanks Derelict. That's the confirmation I was looking for and narrows things down enough for me that all I should have to do is sniff the traffic from that application starting up to get an idea of what it's doing. That is assuming I'm allowed to experiment now that it's "working".
So, just to re-enforce my understanding of things, unless I mess with the bridge filtering settings in system tunables, pfSense will never impact a packet that it isn't routing or NATing, right?
IP traffic on your local subnet is not sent to pfSense. it's sent to the other subnet neighbor directly. Yes, if you bridge, layer 2 traffic will be picked up off the ethernet by pfSense.
Enabling logging on your block rule might be an easy way to see what it's doing.