Can't DHCP from Cable modem
-
OK, spanned a port on the switch and used Wireshark to sniffed the traffic in and out of the WAN interface on the VM host.
I see a DHCP discover leaving my pfSense with the correct MAC address, then immediately see the cable provider reply with a DHCP offer containing my MAC address, but my pfSense never replies with a DHCP request.
Using the pfSense packet capture, I see the DHCP discover leave the server with my MAC address, but never see any reply back with my MAC address. I see all kinds of traffic on the wire coming from the cable including APR's and DHCP offers for other users. Its like others make it in but mine don't.
I can spoof my pfSense MAC address on a XP VM in the same ESXi host, in the ESXi host disconnect the WAN interface from my pfSense VM, connect the WAN interface to the XP machine and it instantly acquires a public IP address from the cable company and the connection works fine to the Internet. This at least verifies my cable modem is not locked to some other MAC.
Any ideas? I will continue searching for a solution.
-
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
-
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
I just tried setting promiscuous mode from "reject" to "accept", didn't make any difference.
I don't understand how VMware could be the problem, WAN1 and WAN3 are on the same vinterface on the same vswitch, the interface VLAN is set "VLAN ID: All (4095)", WAN3 works flawless while WAN1 does not, and WAN2 works flawless though it is static IP.
-
When you tested the connection using the WinXP VM how did you handle the VLANs?
A possibility that occurs to me is that untagged traffic is being sent to WAN1 for some reason. This can cause problems with FreeBSDs VLAN handling that may not appear in WinXP.
You should be bale to see this in packet captures.Can you handle the VLAN untagging in VMware instead?
Steve
-
All I know about VMware and its internals would go into a short paragraph so this reply involves a certain amount of speculation.
Your WAN1 uses non-default MAC address. It has been reported in these forums that some FreeBSD LAN drivers do not correctly program the hardware for a change in MAC address. Perhaps pfSense has not issued the correct incantation to change the MAC address on the WAN1 emulated hardware, consequently VMware doesn't know to deliver the DHCP response (to the non-default MAC address) to the WAN1 interface.
The port monitor in the switch shows the cable modem returning DHCP responses. Can you do packet capture in VMware to verify DHCP responses are coming in off the wire? Can you do packet capture in VMware to verify DHCP responses are delivered to pfSense emulated interface? Can you do packet capture in pfSense to verify DHCP responses are seen on the "physical" interface?
Can you do packet capture in pfSense to verify DHCP responses are seen on the WAN1 VLAN interface?Do you really need the non-default MAC address on WAN1? Could you try the default MAC address? (Might need to reboot for change to take effect.)
When you tested the connection using the WinXP VM how did you handle the VLANs?
A possibility that occurs to me is that untagged traffic is being sent to WAN1 for some reason.Could the switch be configured such that it doesn't ALWAYS add VLAN tags to traffic received from the cable modem?
@!real:
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
I just tried setting promiscuous mode from "reject" to "accept", didn't make any difference.
The words suggest to me that the setting allows or rejects an attempt by the client to set an interface to promiscuous mode (accept ALL frames regardless of the destination MAC address). They don't suggest a setting to switch the interface between promiscuous and "normal" mode. I suspect you MIGHT need to set the vswitch promiscuous mode to accept AND set the pfSense physical interface into promiscuous mode (e.g. by pfSense ifconfig shell command or . . .).
-
if you're MAC spoofing in VMware, you must enable promiscuous mode. That sounds more like the issue described most recently where the DHCP responses are on the physical wire, but aren't seen in the VM, which means they're getting dropped within ESX. ESX will drop traffic destined to anything other than VM's assigned virtual MAC address if the vswitch isn't in promiscuous mode.
-
It would be interesting to see what ESXi thinks is the MAC address of that interface.
If you're comfortable with using SSH into your ESXi host and running a script you could try this:
http://www.virtuallyghetto.com/2011/05/how-to-query-for-macs-on-internal.html
I haven't tried it but I'm pretty sure tcpdump is also included ESXi.
-
OK, thanks to stephenw10 I have worked around the problem. Something is still going wrong but at least I have IP's on all my WAN interfaces for the first time!
What I did was left the Cisco 2960 interface (that all WAN's attach) trunked but set the native VLAN to 900. Then set WAN1 to the parent pfSense interface (non VLAN) and it instantly picked up a public IP address from the cable system.
So basically I'm sending WAN1 packets untagged, while WAN2 and WAN3 packets are still tagged. WAN2 and WAN3 seem to still work fine but have not thoroughly tested that yet.
Will do more testing tomorrow. I'm going to try spoofing the MAC on my other DHCP WAN interface and see if it quits working.
-
Interesting. The setup you have now is exactly what I would be trying to avoid!
Sending VLAN tagged and non-tagged traffic to the same pfSense interface can cause problems. However it doesn't always by any means. If it's working for you…
An interesting result though, as though the tagging/untagging process is causing packets not to get through.
I know pretty much nothing about vmware outside the concepts. I was suggesting it may be possible to create a virtual switch that accepts the trunk from the Cisco switch and untags the VLAN traffic to three interfaces internally which then feed pfSense. That way you remove the VLAN handling from pfSense.
Steve
-
I know pretty much nothing about vmware outside the concepts. I was suggesting it may be possible to create a virtual switch that accepts the trunk from the Cisco switch and untags the VLAN traffic to three interfaces internally which then feed pfSense. That way you remove the VLAN handling from pfSense.
Steve
Steve, this can easily be done in VMware. I "thought" it would have been easier to just trunk everything on a single interface but has turned out not the case. From what I have seen so far separate interfaces and no tagging seems the best solution. At work I have over 40 virtual servers (mostly Microsoft :-[) and have never had any problem like this with VMware before.
-
There is an 'issue' with FreeBSD and VLAN handling that can cause a problem when you have tagged and untagged traffic on the same interface. I am not familiar with it and I have never run into it myself but I know it exists. It shouldn't be a problem for you since originally you only had tagged traffic on the interface. In another thread I was trying to diagnose a problem and another user found that his switch was sending untagged packets to his pfSense box accidentally. To eliminate that as a possible cause of trouble I was suggesting you move VLAN handling away from pfSense.
However since you are now definitely sending untagged traffic, and it's working, I'm not sure what to suggest! :-\Steve
Edit: Out of interest I looked further into this and can find nothing so now I'm unsure where this advice came from or what it applies to. Certainly general advice within pfSense is to avoid tagged and untagged traffic on the same interface but whether that is due to FreeBSD or VLANs in general I don't know.
-
OK, problem is something to do with MAC spoofing. I spoofed the MAC on WAN3 and it stopped working. Even after clearing the MAC spoofing box it still would not work until a system reboot which got it working again non-spoofed.
One thing I did notice that was odd, while both WAN1 and WAN2 were spoofed (and not working) ifconfig in pfSense down at the bottom showed the VLAN'ed interfaces with their spoofed MAC's, but oddly WAN2 (which I have never spoofed) showed the same MAC as the spoofed WAN1 MAC. WAN3 showed it's last spoofed MAC even though at this time the spoofing box was cleared on the interface web page. After the reboot everything looked as expected, WAN1 showed it's spoofed MAC, while WAN2 and WAN3 showed the VMware interface MAC.
The problems seems to be spoofing and tagging don't work together. I can spoof non-tagged, and can tag non-spoofed, but can NOT spoof tagged.
Anyone else spoofing and tagging together?
-
you can do any of that, just that VMware complicates things because of its MAC restrictions. When you remove MAC spoofing it doesn't go back to the hardware MAC because there is no way to determine what that MAC is until you reboot. When you spoof on a VLAN parent NIC, it sets that for all the VLANs because of other complications with how that functions and the way many NICs work.
-
Why can't you use the default (hardware) MAC address on WAN1?
-
Why can't you use the default (hardware) MAC address on WAN1?
I could, but that is not what I want to do.
In the past I have always used spoofed the same easy to remember MAC towards the cable modem. This way when I want/need to switch internal devices I can simply shut down one, spoof the same MAC and bring up the other device anywhere on the cable modem VLAN without having to go to and rebooting the cable modem time and time again.
-
My experience was MAC spoofing and VLAN tagging does not work together. To work around my problem I set the spoofed MAC address that I wanted my cable modem to see from my WAN interface, inside the VM setup inside VMware and removed the spoofed MAC address from pfSense. This way pfSense sees and uses my spoofed MAC at boot time as if it was a MAC address on a physical NIC. My cable modem sees and locks to my spoofed MAC and all 3 WAN interfaces works correctly on separate VLAN's on the same physical interface.
This setup has been working fine for a week now. I finally have IP's on all three WAN interfaces with the cable modem locked to the MAC address I need it to use.
Thanks to everyone for all your suggestions!
