PfSense Filtering Bridge on Wrong Interfaces?

asayler

Removing the VLAN from the LAN_WIFI member interface doesn't seem to have any effect:


LAN_WIRE (opt2) -> lagg0_vlan3 -> None
LAN_WIFI (opt4) -> igb4        -> None
LAN (opt5)      -> bridge0     -> v4: 192.168.9.1/24

BUT interesting note:

I didn't realize this before, but it appears that traffic flowing across the bridge in the LAN_WIRE -> LAN_WIFI direction is logged and filtered correctly on the LAN_WIRE interface. It's only traffic in the opposite direction (e.g. LAN_WIFI -> LAN_WIRE) which skips the LAN_WIFI filters and jumps straight to the LAN filters…

Which seems strange, since at this point, the LAN_WIFI interface is just a basic igb interface (no LAGG, no VLAN, etc). So of teh two, its the one I would expect to work...

I'll poke at it some more tomorrow, but I'm still open to other suggestions if anyone has them.

stephenw10

Obvious question is what rules do you have on LAN_WIFI?

asayler

@stephenw10:

Obvious question is what rules do you have on LAN_WIFI?

Pretty simple: both LAN_WIFI and LAN_WIRE have the same set of rules: "pass DNS", "pass NTP", and then "pass all with logging".

The LAN bridge interface just has a single "pass all with logging" rule.

Traffic from LAN_WIRE gets logged by the LAN_WIRE "pass all" rule (as expected), but traffic from LAN_WIFI gets logged by the LAN "pass all " rule (should be LAN_WIFI).

asayler

So I setup another test to simplify this a bit:


TEST1 (igb4)
  |
  |---TEST (Bridged: TEST1 + TEST2)
  |
TEST2 (igb5)

With the following rules:


TEST1: Pass All IPv4 with Logging
TEST2: Pass All IPv4 with Logging
TEST:  Deny All IPv4 with Logging

And the bridging system tunables still set to their default config (filter on member interfaces, not bridge interface).

Some behavior as before:


PING TEST2 -> TEST1 - Passed,  Logs as TEST2 Pass All
PING TEST1 -> TEST2 - Blocked, Logs as TEST  Deny All

igb4 has been associated with the problem interface in both cases, so maybe I'll try again with a different interface in case it's a quirk of igb4.

stephenw10

So that's just the real interfaces, no VLANs or LAGG?
If you're pinging between the bridge members why is the traffic hitting the bridge interface at all? It should go directly.
Are the NIC MACs being correctly read?

Steve

asayler

The ports are are igb physical NICs.

I'm pinging between a host connected to the TEST1 interface and a host connected to the TEST2 interface. Those pings should cross the bridge (since that's what a bridge does), but the filtering should happen at the TEST1 and TEST2 interfaces, not the bridge interface itself.


igb4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=503bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:42
	inet6 fe80::290:bff:fe33:e242%igb4 prefixlen 64 scopeid 0x5 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether 00:90:0b:33:e2:43
	inet6 fe80::290:bff:fe33:e243%igb5 prefixlen 64 scopeid 0x6 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	ether 02:fe:4a:c8:9c:00
	inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255 
	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: igb5 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 6 priority 128 path cost 55
	member: igb4 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 5 priority 128 path cost 55</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>

stephenw10

Are those NICs not identical, they have consecutive MACs?
igb4 has VLAN_HWFILTER enabled and igb5 doesn't. No idea what that might be doing now you're not using VLANs.
If the bridge is working correctly I wouldn't expect to see any traffic hit the rules on the bridge interface because it's not being routed to some other interface. The traffic between the two hosts should remain on the bridge only. So why is it both hitting the bridge interface filters at all and not the member interface filters. :-\

Steve

asayler

@stephenw10:

Are those NICs not identical, they have consecutive MACs?

They're constitutive NICs on the same mobo: it's pretty common for such nics to have consecutive MACs

@stephenw10:

igb4 has VLAN_HWFILTER enabled and igb5 doesn't. No idea what that might be doing now you're not using VLANs.

Not sure what's going on there. Maybe leftover from previous vlan setups on igb4…

@stephenw10:

If the bridge is working correctly I wouldn't expect to see any traffic hit the rules on the bridge interface because it's not being routed to some other interface. The traffic between the two hosts should remain on the bridge only. So why is it both hitting the bridge interface filters at all and not the member interface filters. :-\

Yep, that's the problem this thread is about. Why is traffic hitting the bridge interface filter instead of the member interface filter? Hence the title: "pfSense Filtering Bridge on Wrong Interfaces?" ::)

It works correctly in one direction, but not the other. pfSense has tunable options to toggle between filtering at the bridge interface or filtering at the member interfaces, but member interface filtering is the default and that's what I have setup. And I don't think it's ever suppose to change behavior based on direction: the system tunable controlling bridge vs member filtering should be consistent regardless of direction.

stephenw10

Yes, just thinking out loud there.

@asayler:

it's pretty common for such nics to have consecutive MACs

Exactly, which is why I thought they were probably identical yet are reporting different hardware options.

Do they report the same capabilities in 'ifconfig -m igb4'?

Steve

asayler

@stephenw10:

Do they report the same capabilities in 'ifconfig -m igb4'?

I believe the VLAN_HWFILTER difference was just because I still had a VLAN setup on the igb4 interface, it just wasn't assigned anywhere. I deleted the VLAN and rebooted and now it's gone:


[2.2-BETA]/root(1): ifconfig -m igb4
igb4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>capabilities=507fb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:42
	inet6 fe80::290:bff:fe33:e242%igb4 prefixlen 64 scopeid 0x5 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	supported media:
		media autoselect
		media 1000baseT
		media 1000baseT mediaopt full-duplex
		media 100baseTX mediaopt full-duplex
		media 100baseTX
		media 10baseT/UTP mediaopt full-duplex
		media 10baseT/UTP</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>


[2.2-BETA]/root(2): ifconfig -m igb5
igb5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>capabilities=507fb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:43
	inet6 fe80::290:bff:fe33:e243%igb5 prefixlen 64 scopeid 0x6 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	supported media:
		media autoselect
		media 1000baseT
		media 1000baseT mediaopt full-duplex
		media 100baseTX mediaopt full-duplex
		media 100baseTX
		media 10baseT/UTP mediaopt full-duplex
		media 10baseT/UTP</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>

Let me dig into it a bit more this evening. I may have stumbled on something but I need to poke at it some more first…

asayler

I played with this a bit more. It appears that maybe I'm just misunderstanding the intended bridging behavior. Here's how the system appears to be behaving now:

TEST1 (igb4)
|
|–- TEST_BRIDGE (TEST1 + TEST2)
|
TEST2 (igb5)

TEST1: Pass All IPv4 with Logging
TEST2: Pass All IPv4 with Logging
TEST_BRIDGE: Pass All IPv4 with Logging

PING TEST1 -> TEST2 - Logs as TEST1 Pass All
PING TEST2 -> TEST1 - Logs as TEST2 Pass All
PING TEST1 -> Internet - Logs as TEST_BRIDGE Pass All
PING TEST2 -> Internet - Logs as TEST_BRIDGE Pass All

So it appears as though cross-bridge traffic (e.g. TEST1 to TEST2) get handled by the rules for the ingress member port, whereas traffic from a bridge-port to an external device (e.g. the Internet) gets filtered by the bridge rules.

Is that how its suppose to work? I was assuming that everything would just be filtered by the member interface rules and that the bridge rules wouldn't come into play at all, but maybe that's not so?

stephenw10

That's certainly not how I understood it.
How do you have the sysctls set in the above setup?

It's an interesting point you raise though. The bridge setup I have on my test box has no rules on the member interfaces. Whilst it's never given any trouble, clients on the bridged interfaces can access external addresses and the webgui, I can't be sure whether I ever tested connectivity between the bridged nics. :-
I'll test it when I get a chance.

Steve

asayler

@stephenw10:

How do you have the sysctls set in the above setup?

I believe they're set to the default behavior of only filtering on the member interfaces:

net.link.bridge.pfil_member = 1
net.link.bridge.pfil_bridge = 0