UDP broadcasts to WAN
-
And, real quick, be sure one of your LOCAL VLANs isn't mistakenly created on vr0 instead of your tagged LOCAL interface. It's easy to do.
-
And, real quick, be sure one of your LOCAL VLANs isn't mistakenly created on vr0 instead of your tagged LOCAL interface. It's easy to do.
[2.2-RELEASE][admin@pfSense.localdomain]/root: ifconfig | grep vr0 vr0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 inet6 fe80::20d:b9ff:fe17:cb28%vr0 prefixlen 64 scopeid 0x1 [2.2-RELEASE][admin@pfSense.localdomain]/root:</up,broadcast,running,simplex,multicast>
No, it isn't. Good point, though.
Risto
-
What interface was that capture taken on?
Actually on another host on wan side. But the sender's MAC is pfSense's WAN.
@Derelict:Please provide a few more, captured on the WAN interface. Preferably some generic broadcasts like ARP, DHCP, etc. I don't have any windows CIFS hosts to test with - at least not readily.
I'll try tomorrow.
Risto
-
Well, I just built this with three VLANs. PRIVATE seems to work:
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 02:ba:93:e5:a4:00
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::1:1%bridge0 prefixlen 64 scopeid 0xb
nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: re2_vlan30 flags=943<learning,discover,<strong>PRIVATE,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 55
member: re2_vlan20 flags=943<learning,discover,<strong>PRIVATE,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 55
member: re2_vlan10 flags=943<learning,discover,<strong>PRIVATE,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 55interface FastEthernet0/4
switchport access vlan 10
switchport mode access
!
interface FastEthernet0/5
switchport access vlan 20
switchport mode access
!
interface FastEthernet0/6
switchport access vlan 30
switchport mode access
!
interface FastEthernet0/7
switchport trunk allowed vlan 10,20,30
switchport mode trunk
!</learning,discover,<strong></learning,discover,<strong></learning,discover,<strong></performnud></up,broadcast,running,simplex,multicast>Host on fa0/4 gets DHCP and can surf but is isolated from host on fa0/5, that also gets DHCP in the same subnet. Not so much as ARP. New tool for the toolbox. Might be useful in a pinch if I'm ever on a desert island without newegg and need pvlanedge.
But I am not seeing any broadcasts on the BRIDGE0 network leaking out WAN, which will come as a surprise to absolutely nobody except, perhaps, you. ;)
If I were you, I would just forget that you've found some defect in pfSense and, instead, look at what you've done in your configuration or testing process to see what you think you're seeing.
If you can tell me exactly how to duplicate it, I'd be happy to try. With some effort I can get a windows VM into this test environment.
Even on my lowly Mac I was able to generate some netbios name lookups. The only thing on WAN when I do the same thing is apinger.
![Screen Shot 2015-03-03 at 8.21.25 PM.png](/public/imported_attachments/1/Screen Shot 2015-03-03 at 8.21.25 PM.png)
![Screen Shot 2015-03-03 at 8.21.25 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-03-03 at 8.21.25 PM.png_thumb) -
Now the problem seems to be policy based routing. Maybe in combination with the system tunable settings I mentioned before (but forgot to mention yesterday, sorry): net.link.bridge.pfil_member=0, net.link.bridge.pfil_bridge=1.
You would need a second gateway, create a gateway group, put the gateways on different tiers (the lower number will receive the traffic), and select the group as gateway in the firewall rule, instead of default.
The packet in your attachment should qualify.
Risto
-
Now the problem seems to be policy based routing. Maybe in combination with the system tunable settings I mentioned before (but forgot to mention yesterday, sorry): net.link.bridge.pfil_member=0, net.link.bridge.pfil_bridge=1.
I set mine the same for the previous tests.
You would need a second gateway, create a gateway group, put the gateways on different tiers (the lower number will receive the traffic), and select the group as gateway in the firewall rule, instead of default.
The packet in your attachment should qualify.
Not sure that I'm willing to set that up since I have no reason to believe the results will be any different. You need to take a GOOD look at what you've done in your environment. What you're describing is basically impossible. You screwed something up somewhere. Probably at layer 2.
-
I'm trying to say that this one config change, from default to gateway group, changes the behaviour.
You could simply use any, even imaginary, host on wan side as your second gateway.
ARP or DHCP are not leaking.
Risto
-
How would "routing" anything have to do with it.. What part do you just not understand that BROADCAST traffic is NOT routed.. what you posted is not even a directed broadcast - its full FF.. Why would pfsense send that anywhere, not going to forward it, not going to route it..
-
a reason for everything. In this case I want to separate the vlans (private bridge ports) so they don't see each other. There is no excuse for pfsense's bridge not working as it should.
Dude, you are totally lost. When you want separate private VLANs, then for goddamn sake do NOT bridge them. Plus, the DHCP is the most BS reason to create a bridge, ever. All of this - overengineered, error prone nonsense with multiple additional layers of complexity that may (and clearly do) cause issues - just because you are lazy to get up DHCP server properly.
-
How would "routing" anything have to do with it.. What part do you just not understand that BROADCAST traffic is NOT routed.. what you posted is not even a directed broadcast - its full FF.. Why would pfsense send that anywhere, not going to forward it, not going to route it..
How should a directed broadcast look like?
Risto
-
Dude, you are totally lost. When you want separate private VLANs, then for goddamn sake do NOT bridge them. Plus, the DHCP is the most BS reason to create a bridge, ever. All of this - overengineered, error prone nonsense with multiple additional layers of complexity that may (and clearly do) cause issues - just because you are lazy to get up DHCP server properly.
Thanks for telling me, but actually we've gone through all this before during this thread, so I don't care to explain it anymore, unless you insist. And it's working with a simple firewall setting. I'm more worried about pfsense and possibly the underlying freebsd.
Risto
-
So I looked at what you posted again..
192.168.1.31.137 > 192.168.1.255.137
That is a directed broadcast…. And what IP address is 1.31? Some on your lan side.. In what world would that ever be routed anywhere?? The only way that would go out some interface that was not in that network is if there was a bridge!
Or you have a mask wrong somewhere where that .255 would be a host IP.. like 192.168.1.0/23 But if pfsense was going to route that as a host address, why would it not be natted if going on your wan? What network is your wan on?
What are the networks on your pfsense with masks? What network is the wan in?
-
So I looked at what you posted again..
192.168.1.31.137 > 192.168.1.255.137
That is a directed broadcast…. And what IP address is 1.31? Some on your lan side.. In what world would that ever be routed anywhere?? The only way that would go out some interface that was not in that network is if there was a bridge!
Yes, 1.31 is a windows box on the lan side. Looks to me that somehow this policy based routing overrides the routing table and ignores the local routes. I think it's wrong.
@johnpoz:Or you have a mask wrong somewhere where that .255 would be a host IP.. like 192.168.1.0/23 But if pfsense was going to route that as a host address, why would it not be natted if going on your wan? What network is your wan on?
I use /24 masks for simplicity. The wans are the only exceptions (ethernet dhcp and ppp).
@johnpoz:What are the networks on your pfsense with masks? What network is the wan in?
Wan in vr0. Here ifconfig of all interfaces with ip:
vr0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8280b <rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic,linkstate>ether 00:0d:b9:17:cb:28 inet6 fe80::20d:b9ff:fe17:cb28%vr0 prefixlen 64 scopeid 0x1 inet 80.220.71.201 netmask 0xffffe000 broadcast 80.220.95.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>) status: active vr1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500 options=8280b <rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic,linkstate>ether 00:0d:b9:17:cb:29 inet6 fe80::20d:b9ff:fe17:cb29%vr1 prefixlen 64 scopeid 0x2 inet 192.168.2.7 netmask 0xffffff00 broadcast 192.168.2.255 inet 192.168.0.7 netmask 0xffffff00 broadcast 192.168.0.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>) status: active lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 nd6 options=21 <performnud,auto_linklocal>ural0_wlan0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 ether 00:17:31:c7:8f:6d inet6 fe80::217:31ff:fec7:8f6d%ural0_wlan0 prefixlen 64 scopeid 0x8 inet 192.168.3.7 netmask 0xffffff00 broadcast 192.168.3.255 nd6 options=21 <performnud,auto_linklocal>media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap> status: running ssid pfSense2 channel 1 (2412 MHz 11g) bssid 00:17:31:c7:8f:6d regdomain ETSI country FI authmode WPA privacy MIXED deftxkey 2 TKIP 2:128-bit TKIP 3:128-bit txpower 30 scanvalid 60 protmode OFF dtimperiod 1 -dfs ppp1: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492 inet6 fe80::20d:b9ff:fe17:cb28%ppp1 prefixlen 64 scopeid 0x1e inet 10.233.110.117 --> 10.64.64.1 netmask 0xffffffff nd6 options=21 <performnud,auto_linklocal>bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 ether 02:8f:df:55:b9:00 inet 192.168.1.7 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vr1_vlan120 flags=b63 <learning,discover,private,edge,autoedge,autoptp>ifmaxaddr 0 port 28 priority 128 path cost 200000 ... member: vr1_vlan101 flags=b63 <learning,discover,private,edge,autoedge,autoptp>ifmaxaddr 0 port 9 priority 128 path cost 200000</learning,discover,private,edge,autoedge,autoptp></learning,discover,private,edge,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></up,pointopoint,running,noarp,simplex,multicast></hostap></performnud,auto_linklocal></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,wol_ucast,wol_magic,linkstate></up,broadcast,running,simplex,multicast>
Risto
-
And the switch config? And a physical diagram of how you have it all connected?
-
"Looks to me that somehow this policy based routing overrides the routing table"
That might be something if it wasn't broadcast traffic you don't route broadcast traffic.. So unless you have a mask where that looks like a host IP and not a broadcast address it would not be routed. No matter if policy based or not.
-
And the switch config? And a physical diagram of how you have it all connected?
I haven't succeeded in getting a shell interface to the switch (Dell PowerConnect 2724), so I just have to describe it here. (The web-management has been problematic, too.)
- ports 1 to 20 are untagged with vlans 101 to 120 respectively, these go to apartments
- ports 21 to 23 are without vlans
- port 24 is tagged with vlans 101 to 120, this is connected to pfsense's lan (vr1)
- pfsense's lan has vlans 101 to 120 that comprise bridge0
- pfsense's lan has also two ip addresses for raw access towards the switch
- pfsense's wan (vr0) is connected to operator's line through a switch (another one)
- pfsense's wlan (ural0_wlan0) has an ip address and is used as an alternative access to lan side
- pfsense's 3G stick (ppp1) is used as backup wan
Risto
-
pfsense's wan (vr0) is connected to operator's line through a switch (another one)
What else is connected to that switch?
-
I shouldn't have to create a gateway group to test this. All I'll have to do is policy route to the existing gateway instead of the default routing table.
Even though I know that won't satisfy you so I'll make a group anyway. Not sure how that will satisfy you, either.
-
What else is connected to that switch?
Two more routers.
@Derelict:I shouldn't have to create a gateway group to test this. All I'll have to do is policy route to the existing gateway instead of the default routing table.
Even though I know that won't satisfy you so I'll make a group anyway. Not sure how that will satisfy you, either.
You are probably right. One route is enough. I was thinking too complicated. Sorry for that.
Risto
-
I made a simplified setup with a virtual host in qemu. Two interfaces of type "em" (intel gigabit). Lan ip 192.168.2.1, bridge ip 192.168.0.1 (vlans 101, 102), wan dhcp. I was able to demonstrate the problem by sending a udp packet with nmap. I'll attach the config. It is so simple that it should be easy to spot the error.
-
How do you guys like your crow? I'll take mine with sriracha and a nice zinfandel.
-
Looks to me that somehow this policy based routing overrides the routing table and ignores the local routes.
Of course - that's the entire point of policy routing. In this situation with a bridge, specifying a gateway on pass rules that match broadcast traffic will forward the broadcast traffic. It's what you're telling it to do. Don't match traffic with a pass rule specifying a gateway that you don't want sent to that gateway.
As others have noted, the bridge is possibly undesirable in this circumstance. If it's not, block broadcast destination traffic above any pass rule specifying a gateway that would match, as any matching traffic will be forced to that gateway.
-
Umm. OK. I guess nobody expects that their LAN NETBIOS name lookups, which are directed at the LAN subnet will be sent to WAN without NAT or anything when they enable policy routing.
This doesn't seem like correct behavior. I guess the discussion can change from "is it really doing that" to "Is it proper for it to do that."
To reiterate:
Interface LAN
Interface address: 192.168.1.1/24
Receives broadcast to: 192.168.1.255
Forwards it to another gateway? Why?And, in my testing, it doesn't happen with a normal interface for LAN. Only with a bridge for LAN. (I just tested removing the private flag on the members. Does the same thing.)
-
So I guess this is specific to bridges. More reason not to use them. Said it before and I'll say it again. Put all your apartments on switch ports. pvlan edge or asymmetric VLANs.
With one of these you can do what you want with one VLAN without a bunch of nonsense.
http://www.ebay.com/itm/CISCO-WS-C2950T-48-SI-48-Port-Switch-10-100-Ethernet-Ports-REFURBISHED-/321672574121
48 10/100
2 10/100/1000
Private VLAN Edge
$29 -
http://www.ebay.com/itm/CISCO-WS-C2950T-48-SI-48-Port-Switch-10-100-Ethernet-Ports-REFURBISHED-/321672574121
The shipping costs to Finland are unacceptable, but of course I could take a look at the European eBay offerings.
@Derelict:This doesn't seem like correct behavior. I guess the discussion can change from "is it really doing that" to "Is it proper for it to do that."
I would express this (with my not perfect English) as the implementation not being ideal.
@cmb:Of course - that's the entire point of policy routing. In this situation with a bridge, specifying a gateway on pass rules that match broadcast traffic will forward the broadcast traffic. It's what you're telling it to do. Don't match traffic with a pass rule specifying a gateway that you don't want sent to that gateway.
Is there a situation where this behavior is wanted? Maybe for bridging two remote sites? No, that would not work.
@cmb:As others have noted, the bridge is possibly undesirable in this circumstance. If it's not, block broadcast destination traffic above any pass rule specifying a gateway that would match, as any matching traffic will be forced to that gateway.
That is my current solution.
Risto
-
@cmb:
Of course - that's the entire point of policy routing. In this situation with a bridge, specifying a gateway on pass rules that match broadcast traffic will forward the broadcast traffic. It's what you're telling it to do. Don't match traffic with a pass rule specifying a gateway that you don't want sent to that gateway.
Is there a situation where this behavior is wanted? Maybe for bridging two remote sites? No, that would not work.
Huh? It is required. Otherwise, the policy routing would not work - at all. Has nothing to do with bridging.
-
"specifying a gateway on pass rules that match broadcast traffic will forward the broadcast traffic."
So your saying shove it down this whole no matter what it is if it meets the rule.. Well while that makes sense, what was the rules that he never showed us, etc. So if I said any from lan net use to dest any use gateway X - would that send broadcast traffic? If so that should prob be mentioned somewhere, is it? I wouldn't think that would send broadcast traffic. You never think of broadcast traffic being routed - but in this case where your forcing stuff down a hole.
So you saying if pfsense sees traffic on its interface and it meets that rule that says send to this gateway, it gets shoved down the hole..
I like mine with sriracha as well ;)
-
Huh? It is required. Otherwise, the policy routing would not work - at all. Has nothing to do with bridging.
No, but has to do with routing. And there is routable traffic and unroutable, like broadcast in his case.
Risto
-
As I understand it, the bridge implementation is a (IP) filtering bridge - the filtering has to be enable on the member interfaces or the bridge as a whole or both. And thus anything that looks like IP will be passed through the filtering rules. So an IP-broadcast packet is going to be processed by the rule set and if it matches first a rule with a gateway then it goes to that gateway. If a rule does not have a gateway, then it goes to the routing table, and in the case of IP-broadcast the destination is in the bridge LAN subnet, so it goes out the various bridged interfaces.
I guess the filtering bridge is all good for if you want to put special rules on to block some traffic coming in 1 port from being bridged out to the other bridge ports - making a "not so transparent" bridge which walls off certain things that are still in the same subnet.
But it also leaves little "tricks" like this gateway policy-routing thing that does not come into play on a single-port ordinary LAN.
Also, I guess these bridges are not generic layer-2 bridges that would just learn MAC addresses and forward stuff around based on MAC address like a network switch. If there is DECnet, AppleTalk, you other favorite protocol that is not IP, then an IP-based packet filter is not going to understand that, so presumably those packets are not bridged?My ravings here might clarify something, or they might just confuse.
-
But it also leaves little "tricks" like this gateway policy-routing thing that does not come into play on a single-port ordinary LAN.
So on an ordinary LAN the routing table comes first, and then the firewall, and then again the routing table, unless there is a policy route? Sounds complicated. Why is it different for a bridge?
Risto
-
According to a PM I received, cmb stated that by the time such traffic from a bridge is evaluated, it is not possible to tell if it is a broadcast or not. I take that to mean there is no longer a way to reference the subnet mask of the bridge interface itself.
I see this as something that is a surprise to pretty much everyone. Let everyone know that if they use a bridge to put a wireless card on the same subnet as their LAN, they will leak information out WAN they probably don't expect. In my testing it was netbios hostnames and internal IP addressing schemes.
I was already putting explicit, quick floating rules on interface group V4WANS outbound blocking RFC1918 and alias local_v4_network destinations. Today I will probably add This Firewall (self). I let the interface checkboxes handle WAN in, though a floating rule blocking alias local_network sources wouldn't be a bad idea. But I think there's some automatic spoofing protection already present.
In summary, the following still applies:
-
Don't bridge. Use a separate subnet for multiple interfaces.
-
Don't use a wireless card, get an AP.
-
Don't bridge ethernet, get a switch.
-
And, in this case, get a switch that does what you need to have done at layer 2 - don't use your router to do your switch's job.
-
Just because you can, doesn't mean you should.
-
-
http://www.ebay.com/itm/CISCO-WS-C2950T-48-SI-48-Port-Switch-10-100-Ethernet-Ports-REFURBISHED-/321672574121
The shipping costs to Finland are unacceptable, but of course I could take a look at the European eBay offerings.
Yeah. It was intended to be an example. There are likely cheap, surplus switches all over the world especially if 10/100 station ports are acceptable. That one has gig uplinks.
-
Why am I getting a strange feeling that some people following this conversation hope that it wasn't ever started, and the whole problem had been kept under the carpet? Probably just my imagination.
Well, I'm glad it was. It certainly has opened my eyes :)
And thanks to all of you for your opinions.
Risto
-
I don't get that at all. We told you from the start that your config was convoluted and unnecessary. You're poking around in areas that get very little attention since there are vastly better ways to accomplish the same thing.
-
The shipping costs to Finland are unacceptable, but of course I could take a look at the European eBay offerings.
There are many at Ebay UK. You should be able to find a 48 port for around €1 per port incl. shipping. More expensive than in the US but still very reasonable.
Now that I found them, I'm almost sad that I don't need one…
-
I don't get that at all. We told you from the start that your config was convoluted and unnecessary. You're poking around in areas that get very little attention since there are vastly better ways to accomplish the same thing.
Well, think again. Since bridge and policy routing are both apparently useful features, it's about time to unveil the fact that they should not be mixed in pfSense. Could happen to anyone.
Risto
-
But it also leaves little "tricks" like this gateway policy-routing thing that does not come into play on a single-port ordinary LAN.
So on an ordinary LAN the routing table comes first, and then the firewall, and then again the routing table, unless there is a policy route? Sounds complicated. Why is it different for a bridge?
Risto
On an ordinary LAN packets that are received for an IP broadcast address (e.g. 192.168.1.255 on CIDR 24) have "arrived" by definition when they hit the network stack of every node on the LAN broadcast domain. That includes the pfSense router. The network stack can just hand the data in the packet to whatever application (if any) in the pfSense is listening for such things.
When a bridge is done, the network stack has to act smarter. pfSense is being "itself" and a bridge at the same time, so it should do the ordinary bit above, but it also has to re-broadcast the packet on all other interfaces in the bridge, to make sure that other devices in the bridged broadcast-domain will get a chance to see the packet.
The handy bit is that, even though "bridging" is really a layer 2 thing, it is also doing layer-3 filtering on the packets - if you want to filter what is bridged you can do that.
The annoying bit is that, even though "bridging" is really a layer 2 thing, it is also doing layer-3 filtering on the packets - it applies the darned filtering rules to the packets, which includes policy-routing and any other weird and wonderful parameters you can add to a "pf" rule - and thus for packets that match those sort of rules it does not look like the bridge the user was expecting!
-
On an ordinary LAN packets that are received for an IP broadcast address (e.g. 192.168.1.255 on CIDR 24) have "arrived" by definition when they hit the network stack of every node on the LAN broadcast domain. That includes the pfSense router. The network stack can just hand the data in the packet to whatever application (if any) in the pfSense is listening for such things.
When a bridge is done, the network stack has to act smarter. pfSense is being "itself" and a bridge at the same time, so it should do the ordinary bit above, but it also has to re-broadcast the packet on all other interfaces in the bridge, to make sure that other devices in the bridged broadcast-domain will get a chance to see the packet.
The handy bit is that, even though "bridging" is really a layer 2 thing, it is also doing layer-3 filtering on the packets - if you want to filter what is bridged you can do that.
The annoying bit is that, even though "bridging" is really a layer 2 thing, it is also doing layer-3 filtering on the packets - it applies the darned filtering rules to the packets, which includes policy-routing and any other weird and wonderful parameters you can add to a "pf" rule - and thus for packets that match those sort of rules it does not look like the bridge the user was expecting!
Very good explanation. Even I understood (I hope). Gives me a thought: if we know that all bridge ports are private (maybe this 'bridge' should be called something else), there is nowhere to forward to, and the whole bridging phase could be skipped.
Risto
-
For the record: I got a Cisco 2950. It has its benefits in my setup, no doubt about that.
Risto