Virtual pfsense - vlan Trunking
I recently configured pfsense on ESXi5, this is a multi WAN setup, the idea is to create seperate vlans for each ISP interface, there will be 12 in total but for now I'm attempting to configure two, VLAN2 and VLAN13. I've read everything I can find on the matter but have some issues.
Option 1, first off if I create a port group for each vlan on the vswitch, and add a vnic for each vlan then it works fine, however there is a mximum limitation in ESXi of 10 vnics per VM, and I need 12.
So option number 2 is to create one ESXi port group and assign it the VLAN number of 4095, which effectively makes it a VLAN trunk. So instead of the vSwitch doing the tagging like in option 1, the vlan tagging is handled by pfsense instead. When using option 2 I can only get one of the two interfaces working, the other interface cannot ping it's upstream gateway.
See pics for details.
I've updated my tplink firmware, I've upgraded my ESXi from 5.5 to 6….
I've got three different model network cards(combo if intel and one Qlogic) in the ESXi...I've tried all three.
pics are listed as option 1 or 2. The config on the tplink switch remains the same on either option.
I'm struggling here as I cannot identify where the problem may lie..if I get the vswith to do the tagging all is well however I have a 10 vnic limitation...
If I get pfsense to do the tagging then i can only ever get one interface working, and it's random, if I rebuild the pfsense config from scratch and set it up with the interface in vlan13 first then this will work fine and the other vlan2 won't...redo it again and create/build with vlan2 fist and then add vlan13 later and vlan2 works and 13 doesn't.
![routing - option 1.jpg](/public/imported_attachments/1/routing - option 1.jpg)
![routing - option 1.jpg_thumb](/public/imported_attachments/1/routing - option 1.jpg_thumb)
![esxi-trunk - option2.jpg](/public/imported_attachments/1/esxi-trunk - option2.jpg)
![esxi-trunk - option2.jpg_thumb](/public/imported_attachments/1/esxi-trunk - option2.jpg_thumb)
![routing - option2.jpg](/public/imported_attachments/1/routing - option2.jpg)
![routing - option2.jpg_thumb](/public/imported_attachments/1/routing - option2.jpg_thumb)
i've only done this for lan connections, but i doubt theres much of a difference … this generally just works.
try packet captures to figure out where it gets stuck
All I get is this "18:36:35.144987 ARP, Request who-has 81...193 tell 81...194, length 28"
It's not even getting to the gateway, which is past the esxi vlan switch, it's like it only allows one vlan to be active at a given time, if vlan2 is operational and vlan 13 is not then I remove/disable vlan2 and restart pfsense, when it comes back online vlan 13 is working, I then enable vlan2 again and it's no longer operational…
I'm wondering if the mtu might have anything to do with it, the only difference between my two options is that option 2 has pfsense doing the vlan tagging, option 1 which works great is doing the vlan tagging on the vswitch.
any other advice or something else I can try?
My next step will be to purchase a test PC and have pfsense running as a physical appliance instead of a virtual....
what type of virtual-nics are you using? in case of vmxnet3, did you install vmwaretools ? if yes, reinstall from scratch without vmware-tools.
other then that, only thing i can think of, is some sort of switch-config error.
could you try to setup that vlan-trunk for LAN, and check if that works? (that way you can do packet-captures from both ends, from pfsense & from windows by wireshark)
I've tried all vnics possible, e1000, vmnex2, vmnex3, with the open-vm-tools package installed.
I've now tried to dedicate a NIC through the esxi as a passthrough device directly to the VM…which means no vswitch to traverse....still exactly the same problem. I've tried a different type of switch, I've configured a Ubiquiti switch, with tagging on the "trunk port" and untagged on the outgoing, see attached, I've also tried port 1 in "trunk" mode directly instead of tagging per vlan, makes no difference, so I'm stumped, how can it be the exact same issue after I've removed the vswitch from the equation, tried a different switch brand.
One vlan only every works, and if I disable the working vlan and power off/on the pfsense VM then the previously non working vlan begins to work. It could still be the ESXi that's causing the issue somehow but I'm finding this difficult to understand since I've used NIC hardware passthough which is like attaching the physical NIC directly to the VM....
Have a look at my ARP table form pfsense, why does pfsense create the two vlans with the same MAC as the parent interface?
And the suspicious thing is that the two different gateways have the same MAC as well, can I make pfsense create/generate different MAC's for the seperate vlan interfaces?
Ok looks like my provider doesn't like seeing the same MAC going to it's devices, I found this which explains pretty well. https://forum.pfsense.org/index.php?topic=42598.0
Unfortunately setting the vlans parent nic to promisc mode does not help me. My parent nic is also EM0, so I installed the shellcmd and did exactly as described with no success.
I also tried to spoof the vlan's MAC address with bogus addresses but this too does not work.
Next thing I could do would be to make static MAC entries for the ISP's gateways in the ARP table, but again the only info I can find to do this would be through DHCP which my interfaces do not use….
Am I destined to fail at achieving this?
Just to add to this, see attached, this is my uplink vlan switch, the arp table is showing the MAC for my WAN parent interface instead of my child VLAN interfaces, which have spoofed MAC's configured.
What else can I try here? I really don't feel like buying a seperate PC and 4x quad nics. There must be a way for the vlan interface to use it's spoofed MAC instead of the parent interfaces MAC.
oh, so you are using 2 vlans to fetch a lease from the same ISP ?
afaik there is no way to change vlan_mac by using the webgui.
see stale feature_request: https://redmine.pfsense.org/issues/2859
Thanks Heper, yes the circuits are all from the same ISP,
you've helped me understand the issue, I know what I need to do to get around this, esxi only allows 10 maximum vnics, but I can also add passthrough devices which it does not count towards it's maximums, so I will use one vswitch with 10 vnics ports, each in its own vlan, and then I will install a dual port nic and do passthrough(i've already done one nic) so will end up with my required 12 and no vlans done through pfsense.