Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfSense Filtering Bridge on Wrong Interfaces?

    Scheduled Pinned Locked Moved
    2.2 Snapshot Feedback and Problems - RETIRED
    2
    19
    3.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      asayler
      last edited by

      Hi All,

      First, I should note that I've only ever tried bridging in pfSense 2.2, so I'm not 100% sure if this is a 2.2 issue, or if it's something more fundamental.

      I've setup pfSense to bridge my Wired LAN port to my Wireless LAN port:

      
LAN_WIRE
  |
  |---LAN (Bridged)
  |
LAN_WIFI

      It's my understanding (from the pfSense 2.1 book, etc) that of these three interfaces, pfSense should be performing filtering on the member interfaces by default, not the bridge interface itself. Yet, when I look at my Firewall logs, all of the traffic seems to be hitting a pass_all rule I added to the bridged LAN interface, not any of the more nuanced rules I've added for either of the two member interfaces:

      
...
Oct 14 00:17:50 - LAN - Allow Any to All - 192.168.9.233:50851 - 233.255.255.250:1990 - UDP
Oct 14 00:17:48 - LAN - Allow Any to All - 192.168.9.19:51470  - 23.245.227.14:123    - UDP
Oct 14 00:17:44 - LAN - Allow Any to All - 192.168.9.13:10761  - 192.168.9.5:123      - UDP
Oct 14 00:17:40 - WAN - Allow SSH        - 75.123.33.113:54405 - 75.144.123.234:22    - TCP:S
Oct 14 00:17:14 - LAN - Allow Any to All - 192.168.9.13:5634   - 192.168.9.5:123      - UDP
...

      The system tunables that control whether filtering occurs on the bridge interface or on the member interfaces are set to their default values:

      
net.link.bridge.pfil_member - Set to 0 to disable filtering on the incoming and outgoing member interfaces. - default (1)
net.link.bridge.pfil_bridge - Set to 1 to enable filtering on the bridge interface                          - default (0)

      Any idea why the system seems to be filtering at the bridge interface instead of the member interfaces? Is this a bug in 2.2, or something in my setup?

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        The sysctls are only read when the bridge interface is created, at boot or otherwise, have you ever switched them?
        Can you show the interface assignments?

        Steve

        1 Reply Last reply Reply Quote 0
        • A
          asayler
          last edited by

          I have rebooted the firewall multiple time to no avail. And I've never touched or changed the relevant sysctl settings: they've always been set to the defaults.

          I'm running the pfSense 2.2-BETA-pfSense (amd64) snapshot from 10/10/14. The initial install dates back to an early version for pfSense 2.2-ALPHA and has been upgraded weekly since then.

          Relvent interfaces:

          
LAN_WIRE (opt2) -> lagg0_vlan3 -> None
LAN_WIFI (opt4) -> igb4_vlan3  -> None
LAN (opt5)      -> bridge0     -> v4: 192.168.9.1/24

          Which translate to:

          
LAN_WIRE -> lagg0_vlan3 -> lagg0               -> igb2 + igb3
LAN_WIFI -> igb4_vlan3  -> igb4
LAN      -> bridge0     -> LAN_WIRE + LAN_WIFI

          And in ifconfig:

          
igb2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 00:90:0b:33:e2:40
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 00:90:0b:33:e2:40
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=503bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:42
	inet6 fe80::290:bff:fe33:e242%igb4 prefixlen 64 scopeid 0x5 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lagg0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether 00:90:0b:33:e2:40
	inet6 fe80::290:bff:fe33:e240%lagg0 prefixlen 64 scopeid 0xb 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
	status: active
	laggproto lacp lagghash l2,l3,l4
	laggport: igb3 flags=1c <active,collecting,distributing>laggport: igb2 flags=1c <active,collecting,distributing>lagg0_vlan3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=303 <rxcsum,txcsum,tso4,tso6>ether 00:90:0b:33:e2:40
	inet6 fe80::290:bff:fe33:e240%lagg0_vlan3 prefixlen 64 scopeid 0xc 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
	status: active
	vlan: 3 vlanpcp: 0 parent interface: lagg0
igb4_vlan3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=303 <rxcsum,txcsum,tso4,tso6>ether 00:90:0b:33:e2:42
	inet6 fe80::290:bff:fe33:e242%igb4_vlan3 prefixlen 64 scopeid 0x10 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 3 vlanpcp: 0 parent interface: igb4
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	ether 02:fe:4a:c8:9c:00
	inet 192.168.9.1 netmask 0xffffff00 broadcast 192.168.9.255 
	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: igb4_vlan3 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 16 priority 128 path cost 2000000
	member: lagg0_vlan3 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 12 priority 128 path cost 55</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,promisc,simplex,multicast></active,collecting,distributing></active,collecting,distributing></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>
          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Hmm, interesting. I don't believe the bridging behaviour has changed between 2.1.X and 2.2. Certainly my test box was upgraded to 2.2 and the bridged interfaces behaved exactly as before.
            I've never tried bridging a LAGG interface. I've never tried having the same VLAN on multiple interfaces.
            I'm going to suggest that because you have VLAN3 present on both bridge members traffic is inherently allowed across. Try disabling the bridge and testing. Try running a different VLAN on igb4. I don't know if this is the expected behaviour or not.  :-
            I await comment from higher authority.  ;)

            Steve

            1 Reply Last reply Reply Quote 0
            • A
              asayler
              last edited by

              I'll try playing with the vlans tonight, but I would hope that's not the issue: having two interfaces on the same vlan is really no different than having all interfaces on no vlan. Either way I wouldn't expect bsd to automatically forward traffic between the interfaces since it's a router, not a switch. That's what the bridging is for…

              Thanks for the input. I'll stay tuned for others who might have more advanced setups and report back when I gather more info.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Yes, I agree, thinking about it more I would not expect that behaviour. However it looks like the most likely suspect here. At least somewhere to start poking for bugs.

                Steve

                1 Reply Last reply Reply Quote 0
                • A
                  asayler
                  last edited by

                  Removing the VLAN from the LAN_WIFI member interface doesn't seem to have any effect:

                  
LAN_WIRE (opt2) -> lagg0_vlan3 -> None
LAN_WIFI (opt4) -> igb4        -> None
LAN (opt5)      -> bridge0     -> v4: 192.168.9.1/24

                  BUT interesting note:

                  I didn't realize this before, but it appears that traffic flowing across the bridge in the LAN_WIRE -> LAN_WIFI direction is logged and filtered correctly on the LAN_WIRE interface. It's only traffic in the opposite direction (e.g. LAN_WIFI -> LAN_WIRE) which skips the LAN_WIFI filters and jumps straight to the LAN filters…

                  Which seems strange, since at this point, the LAN_WIFI interface is just a basic igb interface (no LAGG, no VLAN, etc). So of teh two, its the one I would expect to work...

                  I'll poke at it some more tomorrow, but I'm still open to other suggestions if anyone has them.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Obvious question is what rules do you have on LAN_WIFI?

                    1 Reply Last reply Reply Quote 0
                    • A
                      asayler
                      last edited by

                      @stephenw10:

                      Obvious question is what rules do you have on LAN_WIFI?

                      Pretty simple: both LAN_WIFI and LAN_WIRE have the same set of rules: "pass DNS", "pass NTP", and then "pass all with logging".

                      The LAN bridge interface just has a single "pass all with logging" rule.

                      Traffic from LAN_WIRE gets logged by the LAN_WIRE "pass all" rule (as expected), but traffic from LAN_WIFI gets logged by the LAN "pass all " rule (should be LAN_WIFI).

                      1 Reply Last reply Reply Quote 0
                      • A
                        asayler
                        last edited by

                        So I setup another test to simplify this a bit:

                        
TEST1 (igb4)
  |
  |---TEST (Bridged: TEST1 + TEST2)
  |
TEST2 (igb5)

                        With the following rules:

                        
TEST1: Pass All IPv4 with Logging
TEST2: Pass All IPv4 with Logging
TEST:  Deny All IPv4 with Logging

                        And the bridging system tunables still set to their default config (filter on member interfaces, not bridge interface).

                        Some behavior as before:

                        
PING TEST2 -> TEST1 - Passed,  Logs as TEST2 Pass All
PING TEST1 -> TEST2 - Blocked, Logs as TEST  Deny All

                        igb4 has been associated with the problem interface in both cases, so maybe I'll try again with a different interface in case it's a quirk of igb4.

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          So that's just the real interfaces, no VLANs or LAGG?
                          If you're pinging between the bridge members why is the traffic hitting the bridge interface at all? It should go directly.
                          Are the NIC MACs being correctly read?

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • A
                            asayler
                            last edited by

                            The ports are are igb physical NICs.

                            I'm pinging between a host connected to the TEST1 interface and a host connected to the TEST2 interface. Those pings should cross the bridge (since that's what a bridge does), but the filtering should happen at the TEST1 and TEST2 interfaces, not the bridge interface itself.

                            
igb4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=503bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:42
	inet6 fe80::290:bff:fe33:e242%igb4 prefixlen 64 scopeid 0x5 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether 00:90:0b:33:e2:43
	inet6 fe80::290:bff:fe33:e243%igb5 prefixlen 64 scopeid 0x6 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	ether 02:fe:4a:c8:9c:00
	inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255 
	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: igb5 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 6 priority 128 path cost 55
	member: igb4 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 5 priority 128 path cost 55</learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwfilter,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>
                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Are those NICs not identical, they have consecutive MACs?
                              igb4 has VLAN_HWFILTER enabled and igb5 doesn't. No idea what that might be doing now you're not using VLANs.
                              If the bridge is working correctly I wouldn't expect to see any traffic hit the rules on the bridge interface because it's not being routed to some other interface. The traffic between the two hosts should remain on the bridge only. So why is it both hitting the bridge interface filters at all and not the member interface filters.  :-\

                              Steve

                              1 Reply Last reply Reply Quote 0
                              • A
                                asayler
                                last edited by

                                @stephenw10:

                                Are those NICs not identical, they have consecutive MACs?

                                They're constitutive NICs on the same mobo: it's pretty common for such nics to have consecutive MACs

                                @stephenw10:

                                igb4 has VLAN_HWFILTER enabled and igb5 doesn't. No idea what that might be doing now you're not using VLANs.

                                Not sure what's going on there. Maybe leftover from previous vlan setups on igb4…

                                @stephenw10:

                                If the bridge is working correctly I wouldn't expect to see any traffic hit the rules on the bridge interface because it's not being routed to some other interface. The traffic between the two hosts should remain on the bridge only. So why is it both hitting the bridge interface filters at all and not the member interface filters.  :-\

                                Yep, that's the problem this thread is about. Why is traffic hitting the bridge interface filter instead of the member interface filter? Hence the title: "pfSense Filtering Bridge on Wrong Interfaces?"  ::)

                                It works correctly in one direction, but not the other. pfSense has tunable options to toggle between filtering at the bridge interface or filtering at the member interfaces, but member interface filtering is the default and that's what I have setup. And I don't think it's ever suppose to change behavior based on direction: the system tunable controlling bridge vs member filtering should be consistent regardless of direction.

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  Yes, just thinking out loud there.

                                  @asayler:

                                  it's pretty common for such nics to have consecutive MACs

                                  Exactly, which is why I thought they were probably identical yet are reporting different hardware options.

                                  Do they report the same capabilities in 'ifconfig -m igb4'?

                                  Steve

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    asayler
                                    last edited by

                                    @stephenw10:

                                    Do they report the same capabilities in 'ifconfig -m igb4'?

                                    I believe the VLAN_HWFILTER difference was just because I still had a VLAN setup on the igb4 interface, it just wasn't assigned anywhere. I deleted the VLAN and rebooted and now it's gone:

                                    
[2.2-BETA]/root(1): ifconfig -m igb4
igb4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>capabilities=507fb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:42
	inet6 fe80::290:bff:fe33:e242%igb4 prefixlen 64 scopeid 0x5 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	supported media:
		media autoselect
		media 1000baseT
		media 1000baseT mediaopt full-duplex
		media 100baseTX mediaopt full-duplex
		media 100baseTX
		media 10baseT/UTP mediaopt full-duplex
		media 10baseT/UTP</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast> 

                                    
[2.2-BETA]/root(2): ifconfig -m igb5
igb5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>capabilities=507fb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 00:90:0b:33:e2:43
	inet6 fe80::290:bff:fe33:e243%igb5 prefixlen 64 scopeid 0x6 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	supported media:
		media autoselect
		media 1000baseT
		media 1000baseT mediaopt full-duplex
		media 100baseTX mediaopt full-duplex
		media 100baseTX
		media 10baseT/UTP mediaopt full-duplex
		media 10baseT/UTP</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,polling,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>

                                    Let me dig into it a bit more this evening. I may have stumbled on something but I need to poke at it some more first…

                                    1 Reply Last reply Reply Quote 0
                                    • A
                                      asayler
                                      last edited by

                                      I played with this a bit more. It appears that maybe I'm just misunderstanding the intended bridging behavior. Here's how the system appears to be behaving now:

                                      TEST1 (igb4)
                                        |
                                        |–- TEST_BRIDGE (TEST1 + TEST2)
                                        |
                                      TEST2 (igb5)

                                      TEST1: Pass All IPv4 with Logging
                                      TEST2: Pass All IPv4 with Logging
                                      TEST_BRIDGE: Pass All IPv4 with Logging

                                      PING TEST1 -> TEST2 - Logs as TEST1 Pass All
                                      PING TEST2 -> TEST1 - Logs as TEST2 Pass All
                                      PING TEST1 -> Internet - Logs as TEST_BRIDGE Pass All
                                      PING TEST2 -> Internet - Logs as TEST_BRIDGE Pass All

                                      So it appears as though cross-bridge traffic (e.g. TEST1 to TEST2) get handled by the rules for the ingress member port, whereas traffic from a bridge-port to an external device (e.g. the Internet) gets filtered by the bridge rules.

                                      Is that how its suppose to work? I was assuming that everything would just be filtered by the member interface rules and that the bridge rules wouldn't come into play at all, but maybe that's not so?

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        That's certainly not how I understood it.
                                        How do you have the sysctls set in the above setup?

                                        It's an interesting point you raise though. The bridge setup I have on my test box has no rules on the member interfaces. Whilst it's never given any trouble, clients on the bridged interfaces can access external addresses and the webgui, I can't be sure whether I ever tested connectivity between the bridged nics.  :-
                                        I'll test it when I get a chance.

                                        Steve

                                        1 Reply Last reply Reply Quote 0
                                        • A
                                          asayler
                                          last edited by

                                          @stephenw10:

                                          How do you have the sysctls set in the above setup?

                                          I believe they're set to the default behavior of only filtering on the member interfaces:

                                          net.link.bridge.pfil_member = 1
                                          net.link.bridge.pfil_bridge = 0

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post