PfSense Filtering Bridge on Wrong Interfaces?



  • Hi All,

    First, I should note that I've only ever tried bridging in pfSense 2.2, so I'm not 100% sure if this is a 2.2 issue, or if it's something more fundamental.

    I've setup pfSense to bridge my Wired LAN port to my Wireless LAN port:

    
    LAN_WIRE
      |
      |---LAN (Bridged)
      |
    LAN_WIFI
    
    

    It's my understanding (from the pfSense 2.1 book, etc) that of these three interfaces, pfSense should be performing filtering on the member interfaces by default, not the bridge interface itself. Yet, when I look at my Firewall logs, all of the traffic seems to be hitting a pass_all rule I added to the bridged LAN interface, not any of the more nuanced rules I've added for either of the two member interfaces:

    
    ...
    Oct 14 00:17:50 - LAN - Allow Any to All - 192.168.9.233:50851 - 233.255.255.250:1990 - UDP
    Oct 14 00:17:48 - LAN - Allow Any to All - 192.168.9.19:51470  - 23.245.227.14:123    - UDP
    Oct 14 00:17:44 - LAN - Allow Any to All - 192.168.9.13:10761  - 192.168.9.5:123      - UDP
    Oct 14 00:17:40 - WAN - Allow SSH        - 75.123.33.113:54405 - 75.144.123.234:22    - TCP:S
    Oct 14 00:17:14 - LAN - Allow Any to All - 192.168.9.13:5634   - 192.168.9.5:123      - UDP
    ...
    
    

    The system tunables that control whether filtering occurs on the bridge interface or on the member interfaces are set to their default values:

    
    net.link.bridge.pfil_member - Set to 0 to disable filtering on the incoming and outgoing member interfaces. - default (1)
    net.link.bridge.pfil_bridge - Set to 1 to enable filtering on the bridge interface                          - default (0)
    
    

    Any idea why the system seems to be filtering at the bridge interface instead of the member interfaces? Is this a bug in 2.2, or something in my setup?


  • Netgate Administrator

    The sysctls are only read when the bridge interface is created, at boot or otherwise, have you ever switched them?
    Can you show the interface assignments?

    Steve



  • I have rebooted the firewall multiple time to no avail. And I've never touched or changed the relevant sysctl settings: they've always been set to the defaults.

    I'm running the pfSense 2.2-BETA-pfSense (amd64) snapshot from 10/10/14. The initial install dates back to an early version for pfSense 2.2-ALPHA and has been upgraded weekly since then.

    Relvent interfaces:

    
    LAN_WIRE (opt2) -> lagg0_vlan3 -> None
    LAN_WIFI (opt4) -> igb4_vlan3  -> None
    LAN (opt5)      -> bridge0     -> v4: 192.168.9.1/24
    
    

    Which translate to:

    
    LAN_WIRE -> lagg0_vlan3 -> lagg0               -> igb2 + igb3
    LAN_WIFI -> igb4_vlan3  -> igb4
    LAN      -> bridge0     -> LAN_WIRE + LAN_WIFI
    
    

    And in ifconfig:

    
    igb2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 00:90:0b:33:e2:40
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    igb3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 00:90:0b:33:e2:40
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    igb4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=503bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:42
    	inet6 fe80::290:bff:fe33:e242%igb4 prefixlen 64 scopeid 0x5 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    lagg0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 00:90:0b:33:e2:40
    	inet6 fe80::290:bff:fe33:e240%lagg0 prefixlen 64 scopeid 0xb 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
    	status: active
    	laggproto lacp lagghash l2,l3,l4
    	laggport: igb3 flags=1c <active,collecting,distributing>laggport: igb2 flags=1c <active,collecting,distributing>lagg0_vlan3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=303 <rxcsum,txcsum,tso4,tso6>ether 00:90:0b:33:e2:40
    	inet6 fe80::290:bff:fe33:e240%lagg0_vlan3 prefixlen 64 scopeid 0xc 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
    	status: active
    	vlan: 3 vlanpcp: 0 parent interface: lagg0
    igb4_vlan3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=303 <rxcsum,txcsum,tso4,tso6>ether 00:90:0b:33:e2:42
    	inet6 fe80::290:bff:fe33:e242%igb4_vlan3 prefixlen 64 scopeid 0x10 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    	vlan: 3 vlanpcp: 0 parent interface: igb4
    bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    	ether 02:fe:4a:c8:9c:00
    	inet 192.168.9.1 netmask 0xffffff00 broadcast 192.168.9.255 
    	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    	member: igb4_vlan3 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 16 priority 128 path cost 2000000
    	member: lagg0_vlan3 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 12 priority 128 path cost 55</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,promisc,simplex,multicast></active,collecting,distributing></active,collecting,distributing></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast> 
    

  • Netgate Administrator

    Hmm, interesting. I don't believe the bridging behaviour has changed between 2.1.X and 2.2. Certainly my test box was upgraded to 2.2 and the bridged interfaces behaved exactly as before.
    I've never tried bridging a LAGG interface. I've never tried having the same VLAN on multiple interfaces.
    I'm going to suggest that because you have VLAN3 present on both bridge members traffic is inherently allowed across. Try disabling the bridge and testing. Try running a different VLAN on igb4. I don't know if this is the expected behaviour or not.  :-
    I await comment from higher authority.  ;)

    Steve



  • I'll try playing with the vlans tonight, but I would hope that's not the issue: having two interfaces on the same vlan is really no different than having all interfaces on no vlan. Either way I wouldn't expect bsd to automatically forward traffic between the interfaces since it's a router, not a switch. That's what the bridging is for…

    Thanks for the input. I'll stay tuned for others who might have more advanced setups and report back when I gather more info.


  • Netgate Administrator

    Yes, I agree, thinking about it more I would not expect that behaviour. However it looks like the most likely suspect here. At least somewhere to start poking for bugs.

    Steve



  • Removing the VLAN from the LAN_WIFI member interface doesn't seem to have any effect:

    
    LAN_WIRE (opt2) -> lagg0_vlan3 -> None
    LAN_WIFI (opt4) -> igb4        -> None
    LAN (opt5)      -> bridge0     -> v4: 192.168.9.1/24
    
    

    BUT interesting note:

    I didn't realize this before, but it appears that traffic flowing across the bridge in the LAN_WIRE -> LAN_WIFI direction is logged and filtered correctly on the LAN_WIRE interface. It's only traffic in the opposite direction (e.g. LAN_WIFI -> LAN_WIRE) which skips the LAN_WIFI filters and jumps straight to the LAN filters…

    Which seems strange, since at this point, the LAN_WIFI interface is just a basic igb interface (no LAGG, no VLAN, etc). So of teh two, its the one I would expect to work...

    I'll poke at it some more tomorrow, but I'm still open to other suggestions if anyone has them.


  • Netgate Administrator

    Obvious question is what rules do you have on LAN_WIFI?



  • @stephenw10:

    Obvious question is what rules do you have on LAN_WIFI?

    Pretty simple: both LAN_WIFI and LAN_WIRE have the same set of rules: "pass DNS", "pass NTP", and then "pass all with logging".

    The LAN bridge interface just has a single "pass all with logging" rule.

    Traffic from LAN_WIRE gets logged by the LAN_WIRE "pass all" rule (as expected), but traffic from LAN_WIFI gets logged by the LAN "pass all " rule (should be LAN_WIFI).



  • So I setup another test to simplify this a bit:

    
    TEST1 (igb4)
      |
      |---TEST (Bridged: TEST1 + TEST2)
      |
    TEST2 (igb5)
    
    

    With the following rules:

    
    TEST1: Pass All IPv4 with Logging
    TEST2: Pass All IPv4 with Logging
    TEST:  Deny All IPv4 with Logging
    
    

    And the bridging system tunables still set to their default config (filter on member interfaces, not bridge interface).

    Some behavior as before:

    
    PING TEST2 -> TEST1 - Passed,  Logs as TEST2 Pass All
    PING TEST1 -> TEST2 - Blocked, Logs as TEST  Deny All
    
    

    igb4 has been associated with the problem interface in both cases, so maybe I'll try again with a different interface in case it's a quirk of igb4.


  • Netgate Administrator

    So that's just the real interfaces, no VLANs or LAGG?
    If you're pinging between the bridge members why is the traffic hitting the bridge interface at all? It should go directly.
    Are the NIC MACs being correctly read?

    Steve



  • The ports are are igb physical NICs.

    I'm pinging between a host connected to the TEST1 interface and a host connected to the TEST2 interface. Those pings should cross the bridge (since that's what a bridge does), but the filtering should happen at the TEST1 and TEST2 interfaces, not the bridge interface itself.

    
    igb4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=503bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:42
    	inet6 fe80::290:bff:fe33:e242%igb4 prefixlen 64 scopeid 0x5 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    igb5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether 00:90:0b:33:e2:43
    	inet6 fe80::290:bff:fe33:e243%igb5 prefixlen 64 scopeid 0x6 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
    	status: active
    bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    	ether 02:fe:4a:c8:9c:00
    	inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255 
    	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    	member: igb5 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 6 priority 128 path cost 55
    	member: igb4 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 5 priority 128 path cost 55</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast> 
    

  • Netgate Administrator

    Are those NICs not identical, they have consecutive MACs?
    igb4 has VLAN_HWFILTER enabled and igb5 doesn't. No idea what that might be doing now you're not using VLANs.
    If the bridge is working correctly I wouldn't expect to see any traffic hit the rules on the bridge interface because it's not being routed to some other interface. The traffic between the two hosts should remain on the bridge only. So why is it both hitting the bridge interface filters at all and not the member interface filters.  :-\

    Steve



  • @stephenw10:

    Are those NICs not identical, they have consecutive MACs?

    They're constitutive NICs on the same mobo: it's pretty common for such nics to have consecutive MACs

    @stephenw10:

    igb4 has VLAN_HWFILTER enabled and igb5 doesn't. No idea what that might be doing now you're not using VLANs.

    Not sure what's going on there. Maybe leftover from previous vlan setups on igb4…

    @stephenw10:

    If the bridge is working correctly I wouldn't expect to see any traffic hit the rules on the bridge interface because it's not being routed to some other interface. The traffic between the two hosts should remain on the bridge only. So why is it both hitting the bridge interface filters at all and not the member interface filters.  :-\

    Yep, that's the problem this thread is about. Why is traffic hitting the bridge interface filter instead of the member interface filter? Hence the title: "pfSense Filtering Bridge on Wrong Interfaces?"  ::)

    It works correctly in one direction, but not the other. pfSense has tunable options to toggle between filtering at the bridge interface or filtering at the member interfaces, but member interface filtering is the default and that's what I have setup. And I don't think it's ever suppose to change behavior based on direction: the system tunable controlling bridge vs member filtering should be consistent regardless of direction.


  • Netgate Administrator

    Yes, just thinking out loud there.

    @asayler:

    it's pretty common for such nics to have consecutive MACs

    Exactly, which is why I thought they were probably identical yet are reporting different hardware options.

    Do they report the same capabilities in 'ifconfig -m igb4'?

    Steve



  • @stephenw10:

    Do they report the same capabilities in 'ifconfig -m igb4'?

    I believe the VLAN_HWFILTER difference was just because I still had a VLAN setup on the igb4 interface, it just wasn't assigned anywhere. I deleted the VLAN and rebooted and now it's gone:

    
    [2.2-BETA]/root(1): ifconfig -m igb4
    igb4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>capabilities=507fb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:42
    	inet6 fe80::290:bff:fe33:e242%igb4 prefixlen 64 scopeid 0x5 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    	supported media:
    		media autoselect
    		media 1000baseT
    		media 1000baseT mediaopt full-duplex
    		media 100baseTX mediaopt full-duplex
    		media 100baseTX
    		media 10baseT/UTP mediaopt full-duplex
    		media 10baseT/UTP</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast> 
    
    
    [2.2-BETA]/root(2): ifconfig -m igb5
    igb5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>capabilities=507fb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:43
    	inet6 fe80::290:bff:fe33:e243%igb5 prefixlen 64 scopeid 0x6 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
    	status: active
    	supported media:
    		media autoselect
    		media 1000baseT
    		media 1000baseT mediaopt full-duplex
    		media 100baseTX mediaopt full-duplex
    		media 100baseTX
    		media 10baseT/UTP mediaopt full-duplex
    		media 10baseT/UTP</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast> 
    

    Let me dig into it a bit more this evening. I may have stumbled on something but I need to poke at it some more first…



  • I played with this a bit more. It appears that maybe I'm just misunderstanding the intended bridging behavior. Here's how the system appears to be behaving now:

    TEST1 (igb4)
      |
      |–- TEST_BRIDGE (TEST1 + TEST2)
      |
    TEST2 (igb5)

    TEST1: Pass All IPv4 with Logging
    TEST2: Pass All IPv4 with Logging
    TEST_BRIDGE: Pass All IPv4 with Logging

    PING TEST1 -> TEST2 - Logs as TEST1 Pass All
    PING TEST2 -> TEST1 - Logs as TEST2 Pass All
    PING TEST1 -> Internet - Logs as TEST_BRIDGE Pass All
    PING TEST2 -> Internet - Logs as TEST_BRIDGE Pass All

    So it appears as though cross-bridge traffic (e.g. TEST1 to TEST2) get handled by the rules for the ingress member port, whereas traffic from a bridge-port to an external device (e.g. the Internet) gets filtered by the bridge rules.

    Is that how its suppose to work? I was assuming that everything would just be filtered by the member interface rules and that the bridge rules wouldn't come into play at all, but maybe that's not so?


  • Netgate Administrator

    That's certainly not how I understood it.
    How do you have the sysctls set in the above setup?

    It's an interesting point you raise though. The bridge setup I have on my test box has no rules on the member interfaces. Whilst it's never given any trouble, clients on the bridged interfaces can access external addresses and the webgui, I can't be sure whether I ever tested connectivity between the bridged nics.  :-
    I'll test it when I get a chance.

    Steve



  • @stephenw10:

    How do you have the sysctls set in the above setup?

    I believe they're set to the default behavior of only filtering on the member interfaces:

    net.link.bridge.pfil_member = 1
    net.link.bridge.pfil_bridge = 0