MAC address spoofing on VLAN's and impressions from a second-try user

Aaargh

I had a little disaster with my Linux-based router yesterday (the root disk died on me and idiot me didn't have a backup) leaving me without internet or TV. When I originally set up this router, about 2 years ago, I had a look at PFSense and couldn't get it to work correctly on my network due to a bug (you could arguably call it a missing feature) in PFSense.

So yesterday, wanting to get something up and running quickly, I decided to give PFSense another go.

My router is build around an Asus P9D-I with integrated i210 NIC's, a Celeron CPU and 4GB of ECC RAM. It's is used as a NAT router for a gigabit fiber connection, as an IGMP proxy for IPTV, as a DHCP server and caching DNS server.

Sadly, I have to say the issue that I had 2 years ago still isn't resolved and as a result it's impossible to run PFSense as a router for my network.

The situation: the link to my ISP is a trunked connection with 2 VLAN's. One VLAN is used for Internet, the other is used for IPTV. From the LAN side, both need to be accessible for IPTV to work.

The problem: On the WAN side, both VLAN's need to use a different MAC address. If I use the same MAC on both VLAN interfaces, only the last one to do DHCP gets network access. On my Linux router, this is easy. It's a single line in the 'interfaces' file to set a different MAC on one of the virtual interfaces and everything works as expected. On PFSense I can enter a MAC address for each virtual interface, but when I bring the interfaces up the second interface to activate overwrites the MAC of the first virtual interface on the same physical network card.

Some other random impressions/thoughts as second-try user

The web interface looks and feels very outdated, I would have expected it to behave more like a web-app instead of each click resulting in a completely new page loading from the server
The actual firewall rules are not very transparent, there seem to be a lot of implicit hidden rules that are not visible in the UI. The way the rules are shown per-interface doesn't tell me how a packet traverses the list of rules and where the floating rules fit into the order of things. I feel it takes too much control away from me and I would prefer every rule to be explicit.
Ultimately, I was/am uncomfortable with the amount of stuff running on PFSense. There is just way too much going on for my taste, like a webserver running PHP. I like the idea of having a user-friendly UI to configure stuff, but it should be optional and not be running on the router itself.

awebster

What you are looking to do isn't supported by the underlying operating system, however, a possible workaround would be to use a second WAN interface, then connect both WAN interfaces to a small switch and plug the ISP's connection into that switch. Voila, you have 2 MAC addresses on 2 interfaces, which you can configure as you see fit.

NogBadTheBad

Yup it changes the parent interfaces and all the vlan interfaces assigned to it.

I wonder if the interface page should even show the option to change mac and "This field can be used to modify ("spoof") the MAC address of this interface.
Enter a MAC address in the following format: xx:xx:xx:xx:xx:xx or leave blank." if it's a VLAN.

I'm going to pop in a redmine.

Aaargh

@awebster:

What you are looking to do isn't supported by the underlying operating system

But it is supported by the underlying hardware, so this is a failure of the underlying OS.

Sure, there are workaround possible but then I would have to buy additional hardware just to do something the existing hardware already supports. It adds more possible points of failure, etc.

What I'm actually doing is going back to Linux and checking back in a few years.

Aaargh

@NogBadTheBad:

Yup it changes the parent interfaces and all the vlan interfaces assigned to it.

It seems weird to me. It's such a basic feature to support multiple virtual interfaces per physical interface. It's used extensively in virtualisation for example, and most NIC's explicitly support having multiple MAC's. IIRC the i210's on my box support up to 16 MAC's.

awebster

But it is supported by the underlying hardware, so this is a failure of the underlying OS.

FreeBSD, the underlying OS, is not pfSense. If you want the underlying OS to support the hardware the way you want it to, I suggest you take that up in the FreeBSD forums.

Aaargh

@awebster:

But it is supported by the underlying hardware, so this is a failure of the underlying OS.

FreeBSD, the underlying OS, is not pfSense. If you want the underlying OS to support the hardware the way you want it to, I suggest you take that up in the FreeBSD forums.

PFSense made the decision to build their stuff on top of FreeBSD. PFSense as a whole is the product they are selling, who supplies their parts is not my concern. All I see as an end-user is that PFSense as a product can not support my fairly trivial use-case. It's also lacking support for other essential features. E.g. no support for fq_codel. It looks to me like PFSense as a router OS is still a bit of a toy and not a real serious product.

dotdash

Dude,
I'm sorry that something that you got completely free didn't work for your home setup. You could just go back to your Linux box and call it a day instead of starting a troll thread. While it would be good to support this, getting dhcp on two vlans has never been a requirement in any business case I've seen. If you think it's a toy, go ahead and use something else, but it works just fine for many others in home and business environments. I don't know why you want to keep checking back when you obviously think the software sucks.

Nullity

@Aaargh!:

@awebster:

But it is supported by the underlying hardware, so this is a failure of the underlying OS.

FreeBSD, the underlying OS, is not pfSense. If you want the underlying OS to support the hardware the way you want it to, I suggest you take that up in the FreeBSD forums.

PFSense made the decision to build their stuff on top of FreeBSD. PFSense as a whole is the product they are selling, who supplies their parts is not my concern. All I see as an end-user is that PFSense as a product can not support my fairly trivial use-case. It's also lacking support for other essential features. E.g. no support for fq_codel. It looks to me like PFSense as a router OS is still a bit of a toy and not a real serious product.

FYI, fq_codel is support will be included in pfSense 2.4, since it is based on FreeBSD 11 which added fq_codel.

http://caia.swin.edu.au/freebsd/aqm/

I understand that you have complaints (who doesn't?) but you seem like you are a bit more focused on finding reasons to complain rather than finding solutions to your problems.

I'd be the first to say that Linux has more modern networking features and if that's what you want/need, there's nothing wrong with choosing Linux, but I am slightly confused with you saying unconstructive, trollish things like pfSense "is still a bit of a toy and not a real serious product".

johnpoz

Can you not just set mac on a vlan like this?

ifconfig vlan0 lladdr fe:e1:ba:d0:84:0e

This would be a very unique case that you would need to do such a thing - but simple test shows it works

[2.4.0-BETA][root@pfsense.local.lan]/root: ifconfig em2_vlan900
em2_vlan900: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 00:50:56:00:00:03
inet6 fe80::250:56ff:fe00:3%em2_vlan900 prefixlen 64 scopeid 0xe
inet 192.168.99.253 netmask 0xffffff00 broadcast 192.168.99.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 900 vlanpcp: 0 parent interface: em2
groups: vlan
[2.4.0-BETA][root@pfsense.local.lan]/root: ifconfig em2_vlan900 lladdr fe:e1:ba:d0:84:0e
[2.4.0-BETA][root@pfsense.local.lan]/root: ifconfig em2_vlan900
em2_vlan900: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether fe:e1:ba:d0:84:0e
inet6 fe80::250:56ff:fe00:3%em2_vlan900 prefixlen 64 scopeid 0xe
inet 192.168.99.253 netmask 0xffffff00 broadcast 192.168.99.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 900 vlanpcp: 0 parent interface: em2
groups: vlan
[2.4.0-BETA][root@pfsense.local.lan]/root:

You would most likely need to do something or that would not survive a reboot. I would just add a new nic if I need two different mac on the wan side of pfsense.</full-duplex></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,simplex,multicast>

jahonix

@johnpoz:

Can you not just set mac on a vlan like this?

Check it yourself … you'll find that only one of those MACs (last or first, I don't remember) gets used on all VLANs.

voxeljorz

@jahonix:

@johnpoz:

Can you not just set mac on a vlan like this?

Check it yourself … you'll find that only one of those MACs (last or first, I don't remember) gets used on all VLANs.

It's the last VLAN MAC address used.

Rai80

I'm facing the same problem with my new ISP. This new ISP delivers internet on VLAN34 and IPTV on VLAN4. Both interfaces request an IP by DHCP. But they need there own MAC address to get on both VLAN's an IP address. To solve this I created a bridge and added 1 member interface. In my case for VLAN4. After creating the bridge you can change the MAC address of the bridge interface.

JKnott

What Linux distro makes it so easy to spoof MACs, let alone with different VLANs? I've been using Linux for over 20 years (currently openSUSE) and don't ever recall such a thing. While it may be possible with ifconfig or other utilities, spoofing the MAC isn't even included in the network settings, in openSUSE. Also, the only difference between a VLAN frame and a native LAN frame is the VLAN tag, which is included in the frame payload that's passed to the NIC. I don't know that there's any way to change the MAC, as that's done in the NIC, not in data fed to it. Has anyone else seen this in Linux or elsewhere?

In looking at the man page for the IF command (replacement for deprecated ifconfig), there is something called Virtual Function (VF), where things such as MAC or VLAN can be set, but no indication that the MAC can be set differently per VLAN.

BTW, is the ifconfig in Linux that different from the one in BSD?

Rai80

For anyone looking or a solution/workaround to this. Add the following in the top section of /etc/inc/interfaces.inc:

mwexec("/sbin/ifconfig igb0 promisc");
mwexec("/sbin/ifconfig igb0.4 promisc");
mwexec("/sbin/ifconfig igb0.4 ether 00:aa:bb:cc:dd:ee");

stephenw10

You have actually confirmed that allows pulling two IPs via DHCP?

Rai80

Yes, I got 2 IP's!

Wikai

@stephenw10 said in MAC address spoofing on VLAN's and impressions from a second-try user:

You have actually confirmed that allows pulling two IPs via DHCP?

@Rai80 said in MAC address spoofing on VLAN's and impressions from a second-try user:

Yes, I got 2 IP's!

I can confirm this works!
I'm now using a single-NIC Intel NUC as my home router, pulling 5 IPs from my ISP via DHCP and even load-balancing over them (yes... my ISP messed up giving me 100/100 per IP :)).
For years I thought this wasn't possible with pfSense, believing it had to be run virtualized in order to pull this off. But here I am, running a single-NIC bare metal.
Thank you @Rai80.

JKnott

@Wikai

Please show the output from ifconfig. If public addresses, feel free to edit the network part.

GertJanT

I am not familiar with the promiscous option.
But what I read from this:
Usually, your network interfaces will only pass the packets they are programmed to pass to your CPU. In promiscuous mode, your network interfaces will catch every single packet it receives on an interface.

So, isn't here a performance downside to this usage?

Wikai

@JKnott
Here you go.
em0.10 - LAN vlan
em0.101-105 WAN vlans

Shell Output - ifconfig
em0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
	options=1209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
	ether e4:ee:1a:xx:xx:xx
	hwaddr c0:3f:d5:xx:xx:xx
	inet6 fe80::xxxx:xxxx:xxxx:xxxx%em0 prefixlen 64 scopeid 0x1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	
em0.10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether e4:ee:1a:xx:xx:xx
	inet6 fe80::xxxx:xxxx:xxxx:xxxx%em0.10 prefixlen 64 scopeid 0x6
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 10 vlanpcp: 0 parent interface: em0
	groups: vlan
	
em0.101: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 1c:b7:2c:xx:xx:xx
	inet6 fe80::xxxx:xxxx:xxxx:xxxx%em0.101 prefixlen 64 scopeid 0x7
	inet xxx.xxx.xxx.8 netmask 0xffffff00 broadcast xxx.xxx.xxx.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 101 vlanpcp: 0 parent interface: em0
	groups: vlan
	
em0.102: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 1a:c9:8e:xx:xx:xx
	inet6 fe80::xxxx:xxxx:xxxx:xxxx%em0.102 prefixlen 64 scopeid 0x8
	inet xxx.xxx.xxx.17 netmask 0xffffff80 broadcast xxx.xxx.xxx.127
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 102 vlanpcp: 0 parent interface: em0
	groups: vlan
	
em0.103: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether 90:1b:0e:xx:xx:xx
	inet6 fe80::xxxx:xxxx:xxxx:xxxx%em0.103 prefixlen 64 scopeid 0x9
	inet xxx.xxx.xxx.47 netmask 0xfffffe00 broadcast xxx.xxx.xxx.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 103 vlanpcp: 0 parent interface: em0
	groups: vlan
	
em0.104: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether fe:b3:33:xx:xx:xx
	inet6 fe80::xxxx:xxxx:xxxx:xxxx%em0.104 prefixlen 64 scopeid 0xa
	inet xxx.xxx.xxx.240 netmask 0xffffff00 broadcast xxx.xxx.xxx.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 104 vlanpcp: 0 parent interface: em0
	groups: vlan
	
em0.105: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether e4:ee:1a:xx:xx:xx
	inet6 fe80::xxxx:xxxx:xxxx:xxxx%em0.105 prefixlen 64 scopeid 0xb
	inet xxx.xxx.xxx.230 netmask 0xfffffe00 broadcast xxx.xxx.xxx.255
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 105 vlanpcp: 0 parent interface: em0
	groups: vlan

JKnott

@Wikai

First off, there's no need to hide the MAC address it's never seen beyond the local LAN. The only instance where you might be worried is when you use IPv6 and MAC based addresses. Other than that, it's irrelevant.

Now, I've noticed some curious things.

You have one (em0.10) that's 192.168.1.1. Why didn't it get a public address?
em0 has no IPv4 address. Why not?
The different VLANs have different subnet mask lengths and broadcast addresses. How is this possible with the same DHCP server?
em0.103 & ,105, the broadcast address does not match the subnet mask.

What happens if you ping those addresses from outside?

Wikai

@JKnott
They're on the WAN side and visible to my ISP (except for em0.10), in other than that I agree ^^ The chances are slim. I know.

em0.10 is my LAN vlan.
It's just the way I set it up. em0.10 is my LAN vlan. They all get untagged in the switch anyway.
Different DHCP servers and subnets. Ask my ISP... :-) I had to release/renew probably a hundred times in order to get each public IP on its separate subnet, in order for the gateway group/load balance to work.
See 3. em0.103 & 105 netmasks are fffffe00, em0.102 ffffff80, em0.101 & 104 ffffff00.

JKnott

@Wikai said in MAC address spoofing on VLAN's and impressions from a second-try user:

Different DHCP servers and subnets. Ask my ISP... :-) I had to release/renew probably a hundred times in order to get each public IP on its separate subnet, in order for the gateway group/load balance to work.

Then it's not stable. If you have a power failure, will you get the same when you boot up?

See 3. em0.103 & 105 netmasks are fffffe00, em0.102 ffffff80, em0.101 & 104 ffffff00.

You have fe.00 on some with broadcast .255 and others ff. If your broadcast is actually x.x.x.255, you mask must be ffffffffff00

Fire up Packet Capture and filter on DHCP. Then disconnect/reconnect the WAN cable and see if you have different MACs for each VLAN. You'll want to download the capture, so that you can examine it with Wireshark.

Wikai

@JKnott

Then it's not stable. If you have a power failure, will you get the same when you boot up?

Yes. I've been running the same setup but on a multi-NIC machine for over a year with no issues.
My ISP reserves the IPs for each MAC address for up to 24 hours I believe. The release/mac-spoof/renew was only for the initial setup.
I carried the spoofed MAC addresses over to my new setup (the NUC) to make sure I didn't have to do it again.

You have fe.00 on some with broadcast .255 and others ff. If your broadcast is actually x.x.x.255, you mask must be ffffffffff00

IP xxx.xxx.7.47
Gateway xxx.xxx.6.1
Netmask 255.255.254.0
Broadcast xxx.xxx.7.255

Fire up Packet Capture and filter on DHCP. Then disconnect/reconnect the WAN cable and see if you have different MACs for each VLAN. You'll want to download the capture, so that you can examine it with Wireshark.

Don't have the time right now. I'll reply again later.

Wikai

@JKnott
Here's a montage of a packet capture on my WAN5 (em0.105) as i disconnect and reconnect the cable between the switch and single NIC.
It seems I've misconfigured something as I'm able to see the DHCP requests of the other vlans on the packet capture of WAN5.
It seems to be working fine, but this shouldn't be happening if the vlans truly were isolated, right?

As you can see in frame 61 the spoofed vlan MAC shows up as source in the request and in frame 64 it's ACK'd.

alt text

Edit: Here's a pic of a speedtest on a site that supports measuring via web sockets. I'm supposed to only have 100/100 :-)
alt text

Rai80

The DHCP request should not be on the same VLAN. From where did you do this capture?

On the primary interface you should see VLAN ID's.

Wikai

@Rai80
Capture was done on one of the WAN vlan interfaces.
I don't even have the primary interface (em0) assigned in pfSense. I'm only using its vlans for my setup (em0.10 LAN, em0.101-105 WANs) which then get untagged in my switch.

I'll try to explain my setup.
Grab a seat. This is probably the weirdest home pfSense setup you'll come across.

I've got a 100/100 Mbps FTTH connection with 5 public IPs through DHCP.
About 2 years ago I was amazed to find out I could actually pull 100/100 Mbps per public IP (probably misconfig on my ISP's part...)
So I built my first pfSense box with a total of 6 Intel NICs (5 WAN, 1 LAN) and setup a WAN load balance, which gave me 500/500 Mbps in applications that support connections over multiple sockets.

I recently got my hands on a cheap Intel NUC. Tiny fella with AES-NI support.
Downside? Only 1 NIC. So I thought I'd give VLANs a go.
I googled for info and people kept saying it'd have to be run virtualized (in order to get unique MACs for each vlan interface).
Then I found your post and decided to give it a try.
So yesterday I installed pfSense on the NUC and played around with VLANs for the first time, and this is the setup I came up with (pic below).

I'm actually not having any issues with it (so far). I just wanted to confirm that your mwexec-method of giving the VLAN interfaces their own MAC addresses does in fact work :-)

JKnott

@Rai80

So I built my first pfSense box with a total of 6 Intel NICs (5 WAN, 1 LAN)

Is that what you were using? I thought you had a single interface with multiple VLANs that were getting their own IPs, something that shouldn't be happening with VLANs on a single interface.

Also, a tip, when you use Wireshark to display the capture, turn on the VLAN ID, so we know which VLAN you're referring to. A sketch would also be useful, so that we know what you're actually doing.

If you're using multiple NICs, then you're doing nothing different than what I do here, when I plug a computer into the 2nd port on my cable modem and get an address.

Wikai

@JKnott
No, that's my old setup.
I've replaced it with a single-NIC Intel NUC using vlans. Finish reading the post :D (There's an image in the spoiler)