Second vlan only works with interface in promiscuous mode [solved - but why?]

rsingh

environment:
pfsense 2.0, setember snapshot and a snap from today at 4am. hardware is a toshiba a200 laptop.
the lone interface has the physical as LAN, logical vlan 100 for pppoe dsl, vlan 200 for dhcp cable.
re0 - LAN (works)
re0_vlan100 - wan dsl (works)
re0_vlan200 - opt1 cable (only works in promisc mode)

problem:
without intervention, LAN and vlan 100 work fine. tcpdump shows the vlan tags are in the packet headers. vlan 200 doesn't work unless tcpdump is running (promisc) or if i run 'ifconfig re0 promisc' to leave it in promiscuous mode until the next reboot. i tried changing the vlan number just incase there was some issue with my switch or config, the problem persisted.

on the tcpdump captures i was see lots of bad checksums on vlan 200 originating from the pfsense laptop. i went to the advanced config, disabled hardware checksum offloading and the checksum issue went away. i read that it was common for realtek nics.

the only thing non-standard occuring on dhcp vlan 200 is the fact that i clone the MAC address to a MAC my cable isp is aware of.

questions:
any idea on why promisc mode is needed for vlan 200 to work?
is it fine to leave the nic in promisc mode (security)?
what's the best way to have the nic go into promisc on bootup?

ifconfig re0
re0: flags=28943 <up,broadcast,running,promisc,simplex,multicast,ppromisc>metric 0 mtu 1500
options=3898 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic>ether 00:16:d4:ff:bf:9d
inet6 fe80::216:d4ff:feff:bf9d%re0 prefixlen 64 scopeid 0x2
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active

dmesg | more
re0: <realtek 10="" 8101e="" 8102e="" 8102el="" 8103e="" pcie="" 100basetx="">port 0x4000-0x40ff mem 0xda000000-0xda000fff irq 18 at device 0.0 on pci5
re0: Using 1 MSI messages
re0: Chip rev. 0x34000000
re0: MAC rev. 0x00000000</realtek></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic></up,broadcast,running,promisc,simplex,multicast,ppromisc>

wallabybob

It has been discussed elsewhere in these forums in the last few days why it is not a good idea to mix VLAN tagged and untagged traffic on the same interface.

Is OPT1 static IP? PPP? DHCP?

Please elaborate on vlan 200 doesn't work unless tcpdump …: VLAN 200 is waiting for IP address? PPP doesn't start up? web pages stall loading? …

What VLAN switch are you using?

@rsingh:

questions:
any idea on why promisc mode is needed for vlan 200 to work?
is it fine to leave the nic in promisc mode (security)?
what's the best way to have the nic go into promisc on bootup?

One of things promiscuous mode will do is allow receipt of packets which don't have your MAC address as the destination MAC address. PERHAPS something is not quite right with the MAC addresses and your ISP is sending packets to the wrong MAC address, or your switch is not doing the right thing. It almost sounds as if a packet is being sent to the wrong MAC address but containing VLAN200 tag. Did you previously use the switch port re0 is connected to for something else? What destination MAC addresses do you see on received frames in the tcdump.
Should be OK.
I'll pass.

rsingh

I'll look into the tag/untag mixing…

OPT1 is DHCP - PPPOE is using on WAN (VLAN 100) which has no problems.

vlan 200 won't get an IP address via DHCP without running tcpdump on re0:
tcpdump -i re0
vlan 200 will work fine with this and pass traffic. If you start a ping, then stop the tcpdump, the pings stop.

The VLAN switch is a netgear gs108t v1, latest firmware.

Regarding your questions about the MAC address, it seems ok:

tcpdump -ei re0_vlan200
21:32:06.778433 00:14:f1:e8:ee:de (oui Unknown) > 00:0c:29:7b:c7:1c (oui Unknown), ethertype IPv4 (0x0800), length 126: xxxx > xxxx: UDP-encap: ESP(spi=0x0eff41f4,seq=0x1b91), length 84
00:0c:29:7b:c7:1c is the spoofed MAC that I set on the OPT1 page in pfsense's web config -

ifconfig re0_vlan200
re0_vlan200: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 00:0c:29:7b:c7:1c

The switch ports currently in use were previously used. Next time I have to reboot the firewall, I'll reboot the swich as well incase something is stuck in there.</up,broadcast,running,simplex,multicast>

rsingh

This guy has the same switch, similar setup: http://forum.pfsense.org/index.php/topic,28379.30.html
Fixed it by not cloning the MAC address as I am currently doing. This is probably the issue. I'll work with my ISP to get around this.

rsingh

That was the problem for me as well. I removed the MAC cloning, ran ifconfig re0 -promisc
Promisc mode is off and everything works. Good to know.

This is what jimp said about using the physical + logical (parent + vlan) in http://forum.pfsense.org/index.php/topic,22921.0.html:

using the parent interface along with VLANs has always been discouraged. It's not good from a security standpoint, and it also breaks other functionality such as Captive Portal.

That said, while it is discouraged, it should probably continue to work [in 2.0?] if possible, since it does on 1.2.x. It would be a POLA violation to break it unnecessarily, and it could lead to machines upgraded in this state being broken/unreachable post-upgrade.

In most (all?) situations, it's better to tag all VLANs on a trunk port and use corresponding VLAN interfaces on pfSense.

I did a search for "vlan" and "spoof". there are lots of interesting results with many people trying to clone a mac on a vlan interface with no traffic. can anyone familiar with vlans on pfsense tell me why spoofed macs are so problematic with vlans?