Help with VLANS in BRIDGE

broonu

Good afternoon everyone,

I'm having a problem that I do not know if it's the normal pfSense behavior.

TL; DR: pfSense only responds ARP to one VLAN in bridge.

I have pfSense 2.4.4-RELEASE (amd64) virtualized in ESXi.
I created a distributed switch with a trunk (eg vlans 2000, 2001 and 2002).
In pfSense I have the WAN, LAN (used in a isolated VLAN for management) and OPT1.
I created in pfSense those VLANs (in OPT1), created a bridge and added all the VLANS in the bridge. I've set up an IP on the bridge to use DHCP server and it works fine, both the VLAN 2000 client and the VLAN 2001 client automatically renew IP, but only one of them can ping the gateway (pfSense). If the VLAN 2000 client pings the IP of the gateway, the VLAN 2001 client can not. In tcpdump I see that while pfSense answers ARP in VLAN 2000, it does not answer in VLAN 2001.

This is normal behavior, am I doing something wrong?
NOTE: In the vcenter switch I have already enabled promiscuous mode, allow MAC change and forged transmits.

strangegopher

Why create vlans when you are just going to bridge them? Is it for multiple wifi networks? It its for firewall rules management then why not create a interface group?

Anyway as for your problem do vlans 2000 and 2001 get ip from same subnet?

broonu

@strangegopher i need to setup a captive portal to serve multiple vlans, and we receive the vlans directly on esxi hosts. each "area" has its own vlan. we need to use same network for DHCP (we're giving out CGNAT IPs). with esxi standard switch, there is a limit of 10 nics per vm, so im working with distributed switch, this way i just need to setup vlans in pfsense and put a trunk interface in VM.

so, i setup IP 100.64.62.253/23 in bridge0, and DHCP is serving IPs from range 100.64.62.1-100.64.63.252. both clients in both vlans 2000 and 2001 are getting IP, and they can ping each other (through my esxi distributed switch). but just one client ping pfsense 100.64.63.253 (bridge IP). it seems if pfsense receives an apr request, they answer just the first and ignores the other vlan requests.

dont know if I made mysel clear, and sorry for the english.

i'll try to send an image to explain.

strangegopher

What are your firewall rules like? Are they restricting ICMP requests? By default the lan side ICMP should work but not sure about cgnat. What is the default gateway and subnet mask on 2nd PC? Have you tried assigning static IP to 2nd PC (take DHCP out of the equation) and see if that works? Is ping the only thing not working? Like can you access pfsense webpage. Have you looked at firewall logs while running ping and see if they are getting blocked?

broonu

@strangegopher dont hink its a firewall issue, because this problem is not relative to VLAN 2000. I mean, if the first client sending pings is on VLAN 2000, then client on VLAN 2001 dont work and vice versa.

PFSENSE BRIDGE IP: 100.64.63.253/23
PC1 IP: 100.64.62.1/23 GW 100.64.63.253
PC2 IP: 100.64.62.2/23 GW 100.64.63.253

Firewall is set to allow any traffic.
Ping was the basic test, but nothing works because the client on second VLAN dont even receive the MAC address of its gateway (pfsense).

strangegopher

I can't think of why it is not working. maybe you can try contacting netgate directly as they might be able to help you out. https://www.netgate.com/company/contact-us.html

johnpoz

Why are you creating a bridge in pfsense? And what does interface limits have to do with anything?

So you have 3 vlans and your trying to run the same IPv4 space on all 3 of them? Borked out of the gate!

broonu

@johnpoz said in Help with VLANS in BRIDGE:

Why are you creating a bridge in pfsense? And what does interface limits have to do with anything?

So you have 3 vlans and your trying to run the same IPv4 space on all 3 of them? Borked out of the gate!
lets say we have 30 places to place hotspot wifi. each of them has its own vlan. we're delivering pppoe with cgnat (100.64.62.0/23) to these places. we're using a bridge just to optimize the ranges to dhcp server, instead of one dhcp server (and one network range) AND one captive portal to each vlan. that way we can route this 100.64.62.0/23 to cgnat box which will do its job. in firewall bridge we deny inter-vlan traffic. this works pretty well and we have 5 boxes in production with routeros, im just trying to do the same thing in pfsense.

johnpoz

@broonu said in Help with VLANS in BRIDGE:

we're using a bridge just to optimize the ranges to dhcp server

Gibberish... That has ZERO to do with multiple vlans.. If you want 1 layer 2 then you use 1 vlan.. Once you create the bridge no matter what vlan ID you put on them if connected to the same bridge then you have 1 layer 2, all the same broadcast domain.

broonu

@johnpoz i dont care about broadcast domain in this scenario. these vlans are reaching us anyway, it's not there to isolate the traffic. i just need a bridge working to only one dhcp server, with only one range of rfc 6598 addresses and only one captive portal. like i said, this works in mikrotik routeros, just trying to do the same thing in pfsense.

Derelict

What he is saying is if you bridge multiple VLANs they are all now the same broadcast domain so why complicate things with multiple VLANs and a bridge? One VLAN does the same thing without the silly bridging.

in firewall bridge we deny inter-vlan traffic.

Ah so there's the rub. I, personally, would not do that. If you do not want the subnets to talk to each other, then it's easy. Just make several, normal layer 3 interfaces, each with a subnet of the 100.64.62.0/23 network. Set up a DHCP server for each interface.

Assign the Captive Portal to all of them (You can select multiple interfaces to be served on one Captive Portal)

Another option would be if ESXi has the equivalent of "port isolation" in its vswitch. I have no idea if it does or doesn't.

broonu

@derelict said in Help with VLANS in BRIDGE:

What he is saying is if you bridge multiple VLANs they are all now the same broadcast domain so why complicate things with multiple VLANs and a bridge? One VLAN does the same thing without the silly bridging.

yah man, i know exactly what he is saying. dont want to discuss the best practices on this. i know its not the "right" thing to do. this just not the point. just want to know if this works or not. it works like a charm in mikrotik, but i dont like the routeros system and trying to move everything to pfsense.

Derelict

If you are not getting ARP on all bridge members then either the bridge is configured wrong or the switch is configured wrong.

https://www.netgate.com/docs/pfsense/book/bridging/index.html

You can consider the VLAN interfaces as separate, internal interfaces for bridging purposes.

Not quite sure how you're isolating inter-vlan traffic with firewall rules when you don't know what address range is going to be on what interface but I might not have the full picture of what you are doing there in your DHCP.

broonu

@derelict said in Help with VLANS in BRIDGE:

If you are not getting ARP on all bridge members then either the bridge is configured wrong or the switch is configured wrong.

https://www.netgate.com/docs/pfsense/book/bridging/index.html

You can consider the VLAN interfaces as separate, internal interfaces for bridging purposes.

Not quite sure how you're isolating inter-vlan traffic with firewall rules when you don't know what address range is going to be on what interface but I might not have the full picture of what you are doing there in your DHCP.

thank you for your help.
well, the vlans is out of our control, this is why i have to handle this way. i have a small range to put in DHCP server (100.64.63.0/23). This traffic is routed to another interface (CGNAT) to a box that do the NAT to internet. I could breake this range to put a /25 per vlan, but it will be a static thing, and if that specific vlan grows above 128 clients it will need to be changed. with the same /23 to all vlans i dont need to worry about running out IP's. so this is the scenario: the router receive like 20-30 vlans, those vlans are bridged, using same dhcp server and same captive portal in bridge. mikrotik has a filter that deny traffic forward inside the bridge itself, it just permits the traffic cross. client on vlan 10 get ip and need to auth the same way client in vlan 20, but the traffic between him is denied.

i messing around to find if this is a pfsense or vmware thing...

broonu

@derelict said in Help with VLANS in BRIDGE:

If you are not getting ARP on all bridge members then either the bridge is configured wrong or the switch is configured wrong.

https://www.netgate.com/docs/pfsense/book/bridging/index.html

You can consider the VLAN interfaces as separate, internal interfaces for bridging purposes.

Not quite sure how you're isolating inter-vlan traffic with firewall rules when you don't know what address range is going to be on what interface but I might not have the full picture of what you are doing there in your DHCP.

Captive portal

Captive portal (Captive Portal) is not compatible with transparent bridging because it requires an IP on the interface being bridged, used to serve the portal contents, and that IP must be the gateway for clients. This means that it is not possible, for example, to bridge LAN and WAN and hope to capture clients with the portal.

This can work when bridging multiple local interfaces to all route through pfSense (e.g. LAN1, LAN2, LAN3, etc). It will work if the bridge interface is assigned, the bridge interface has an IP address, and that IP address is used as the gateway by clients on the bridge. See Swapping Interface Assignments for a procedure to place the IP address on an assigned bridge interface.

This is exactly what im doing, but my interfaces are VLAN interfaces.

Derelict

Look at the Private Ports settings in the pfSense bridging advanced settings and see if that doesn't help you isolate them from each other.

See also the sysctl settings for the bridge pfil. In your case, with the Private Ports working, I would switch the defaults so:

net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=1

That way you only have to worry about rules on the bridge itself, not the member interfaces.

System > Advanced, System Tunables

Derelict

@broonu said in Help with VLANS in BRIDGE:

This is exactly what im doing, but my interfaces are VLAN interfaces.

So if you pcap on the bridge, what shows up for the member interfaces that aren't working?

What about a pcap on the member interface itself?

Bridging VLAN interfaces works fine. I'd look elsewhere for the problem, like the member interfaces not being configured properly to ESXi in the first place.

broonu

@derelict said in Help with VLANS in BRIDGE:

Look at the Private Ports settings in the pfSense bridging advanced settings and see if that doesn't help you isolate them from each other.

See also the sysctl settings for the bridge pfil. In your case, with the Private Ports working, I would switch the defaults so:

net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=1

That way you only have to worry about rules on the bridge itself, not the member interfaces.

System > Advanced, System Tunables

sure, this is how im doing, and its working for isolate the traffic inter-vlan.

broonu

@derelict said in Help with VLANS in BRIDGE:

@broonu said in Help with VLANS in BRIDGE:

This is exactly what im doing, but my interfaces are VLAN interfaces.

So if you pcap on the bridge, what shows up for the member interfaces that aren't working?

What about a pcap on the member interface itself?

Bridging VLAN interfaces works fine. I'd look elsewhere for the problem, like the member interfaces not being configured properly to ESXi in the first place.

With tcpdump i see the ARP Request but pfsense dont send the ARP Reply.
Im going to clear everything and reconfigure from scratch.

broonu

@delerict thank you for your time and help!
it was a vmware misconfiguration, e1000 nic instead of vmxnet3.
now i'm facing another problem: there is a interfaces limit to be added in a bridge?