Can't DHCP from Cable modem
-
Running pfSense 2.0.1 in ESXi 4.1 VMware, I have 3 WAN's on single interface on 3 VLAN's,
WAN1 el0-VLAN900 > DHCP cable modem
WAN2 el0-VLAN100 > Static microwave
WAN3 el0-VLAN200 > DHCP microwaveProblem is pfSense will never get DHCP from the cable modem on WAN1. It DHCP's perfectly from WAN3, and static WAN2 works flawless.
I have powered on and off in every order I can think of. Always power modem off when switching from device to device. All ignore bogon boxes are unchecked. Log shows dhclient is asking for IP but fails with dhclient error "No DHCPOFFERS received." I can do a packet capture on WAN1 and see all the DHCP and ARP requests for everyone else on the cable system. The pfSense charts around 25kbs-35kbs data inbound on the WAN1 interface and appears to be all the broadcast traffic from the cable system.
I can power off modem, plug laptop to modem, power on modem, laptop gets public IP (NOT private IP) and works flawless.
After pfSense tries to get an IP, I can (without powering down the modem) plug laptop into the modem with static IP=192.168.100.5/24 GW=192.168.100.1/24, then access the modem web page 192.168.100.1 and see that it has captured the correct MAC address entered in the psSense WAN1 interface page.
Running SuddenCrap cable, or Suddenlink as they call themselves, cable modem is Cisco 2100 (which this one apparently only works in bridge mode).
Any idea what is going wrong? Could the firewall somehow blocking the DHCP packets on the cable interface? I feel sure it is something simple I'm looking over. ::)
-
@!real:
Could the firewall somehow blocking the DHCP packets on the cable interface?
Have you checked the firewall log? What firewall rules do you have on WAN1?
Could VMware be filtering the response?
-
@!real:
Could the firewall somehow blocking the DHCP packets on the cable interface?
Have you checked the firewall log? What firewall rules do you have on WAN1?
Could VMware be filtering the response?
I see no logs at all for WAN1, log shows WAN3 blocking lots of stuff.
I have no rules under any of the WAN interfaces. WAN3 with no rules DHCP's fine, WAN1 with no rules does not.
I don't think the VMware can be filtering anything. All WAN's are on a single interface on separate VLAN's, same setup for WAN1 and WAN3 and WAN3 works fine.
Doing a packet capture in pfSense and downloading into Wireshark I see my pfSense do 12 DHCP Discover, but never see any DHCP Offers to my pfSense, I do see DHCP offers for others on the network.
Exactly where in the process is the pfSense packet capture taken? Before or after the firewall rules? If a packet is blocked by the firewall will it show up in the packet capture?
-
Not firewall rule related (unless you're putting in some floating rules overriding the default rules). Cable ISPs commonly won't assign a different device an IP, for some just power cycling the modem suffices, for others you have to either spoof the MAC of whatever device had a lease on there previously, or call them. Seriously doubt it's anything other than that based on your description.
-
One other thing came to mind - any MAC your modem sees may be considered a device connected. For instance I have two public IPs via DHCP with my cable provider, and that means they only allow the first two MAC addresses seen. I ran into troubles when I had a managed switch between the cable modem and my firewalls because something the switch was emitting (STP IIRC, it's been a while) was being picked up by the modem as one of my MACs. So the switch was counted as one of my devices even though it wasn't pulling an IP from them. At one point I got someone with a clue on the phone and they upped it to 3 for me so I could actually use two devices, and that got reset at some point and I couldn't get anyone with a clue so ended up just having to put an unmanaged switch in there. If your ESX host is talking on that network (shouldn't unless it has a management IP or similar on that vswitch) or you have a managed switch, you could be hitting the same thing.
-
@!real:
Exactly where in the process is the pfSense packet capture taken? Before or after the firewall rules? If a packet is blocked by the firewall will it show up in the packet capture?
I believe packet capture is before firewall processing on input to pfSense and after firewall processing on output from pfSense.
I presume you have some sort of VLAN capable switch between the WAN1 cable modem and the pfSense VM. Is that correctly configured - for example, is the cable modem port a member of the correct VLAN? Does the switch have a monitor capability (e.g. direct traffic on a VLAN or a port to a 'monitor' port)? If so, you might be able to use your laptop to see traffic on the VLAN of which the cable modem is a member.
Maybe you could get get some help from the ISP tech support: are they seeing your DHCP requests? are they favourably responding to them?
Does the cable modem have some sort of counter you could look at to verify it is getting the DHCP requests from pfSense - e.g. broadcast packets received on the Ethernet interface?
-
@cmb:
One other thing came to mind - any MAC your modem sees may be considered a device connected. For instance I have two public IPs via DHCP with my cable provider, and that means they only allow the first two MAC addresses seen. I ran into troubles when I had a managed switch between the cable modem and my firewalls because something the switch was emitting (STP IIRC, it's been a while) was being picked up by the modem as one of my MACs. So the switch was counted as one of my devices even though it wasn't pulling an IP from them. At one point I got someone with a clue on the phone and they upped it to 3 for me so I could actually use two devices, and that got reset at some point and I couldn't get anyone with a clue so ended up just having to put an unmanaged switch in there. If your ESX host is talking on that network (shouldn't unless it has a management IP or similar on that vswitch) or you have a managed switch, you could be hitting the same thing.
No floating firewall rules. My setup is currently pretty much factory install with the exception of the 2 extra WAN interfaces.
I do have a managed Cisco 2960 that all my WANs go through. I too have had the managed switch blues where it grabbed the only MAC but have solved that a long time ago with other setups. Plus I can view the modem's web page and verify the modem has locked onto my pfSense MAC address and not something else. The modem seems to lock to anything else with just a power cycle, a laptop or WRT54GL router.
Odd thing is I had this same setup working flawlessly several months back on the exact same switches, same server, same modem, same ESXi VMware, until a power failure corrupted everything and pfSense came back up with all new different interface names, forcing a reinstall. I remember having this problem when I built the first setup but can't remember what I did to solve it.
FYI: This config will keep a Cisco switch from taking a MAC from the modem. This will keep the Cisco switch from speaking until spoken too, and has worked great for me in the past.
interface GigabitEthernet0/1
description WAN from Cable Modem
switchport access vlan 900
switchport mode access
no keepalive
no cdp enable
spanning-tree portfast
spanning-tree bpdufilter enable
end -
@!real:
Exactly where in the process is the pfSense packet capture taken? Before or after the firewall rules? If a packet is blocked by the firewall will it show up in the packet capture?
I believe packet capture is before firewall processing on input to pfSense and after firewall processing on output from pfSense.
I presume you have some sort of VLAN capable switch between the WAN1 cable modem and the pfSense VM. Is that correctly configured - for example, is the cable modem port a member of the correct VLAN? Does the switch have a monitor capability (e.g. direct traffic on a VLAN or a port to a 'monitor' port)? If so, you might be able to use your laptop to see traffic on the VLAN of which the cable modem is a member.
Maybe you could get get some help from the ISP tech support: are they seeing your DHCP requests? are they favourably responding to them?
Does the cable modem have some sort of counter you could look at to verify it is getting the DHCP requests from pfSense - e.g. broadcast packets received on the Ethernet interface?
I can span a port on the switch and sniff the traffic. I'm not physically at the switch, its at the other end of a piece of fiber so I have to go to the switch. Not too hard, but not like its here in the basement. That was why I was asking about the pfSense packet capture, trying to figure out if it may be missing something. Probably best to span a port on the switch and watch it to be sure I'm seeing all the packets.
The web interface on this modem is super simple, not much there at all.
I hate the thought of even trying to find any intelligence at their support. Every time I call all I get is "have you tried turning off and back on yet…"
-
@!real:
[Odd thing is I had this same setup working flawlessly several months back on the exact same switches, same server, same modem, same ESXi VMware, until a power failure corrupted everything and pfSense came back up with all new different interface names, forcing a reinstall. I remember having this problem when I built the first setup but can't remember what I did to solve it.
[/quote]
Any chance you have a backup of previous pfSense configuration?Presumably you also lost your VMware config. Is that backed up?
-
@!real:
[Odd thing is I had this same setup working flawlessly several months back on the exact same switches, same server, same modem, same ESXi VMware, until a power failure corrupted everything and pfSense came back up with all new different interface names, forcing a reinstall. I remember having this problem when I built the first setup but can't remember what I did to solve it.
[/quote]
Any chance you have a backup of previous pfSense configuration?Presumably you also lost your VMware config. Is that backed up?
Unfortunately I never backed up the original pfSense install. The ESXi didn't crash, it is the same setup I had before, I just created a new VM. I had first tried to reset my original install to factory defaults but it seemed the OS was hosed as the interface names were all screwy, so I just started over.
-
I would try to verify the DHCP request is at least getting onto the cable plugged into the modem and look for DHCP responses there.
That you created a new VM for pfSense suggests you may not have precisely recreated the VM configuration used by the working pfSense This may not be significant unless the packet capture at the switch (or on the cable to the modem) shows both DHCP request and response.
-
I believe packet capture is before firewall processing on input to pfSense and after firewall processing on output from pfSense.
yes, it's what is on the wire, with the only exception being checksums if you have hardware checksum offloading (in which case the checksums will be 0).
-
OK, spanned a port on the switch and used Wireshark to sniffed the traffic in and out of the WAN interface on the VM host.
I see a DHCP discover leaving my pfSense with the correct MAC address, then immediately see the cable provider reply with a DHCP offer containing my MAC address, but my pfSense never replies with a DHCP request.
Using the pfSense packet capture, I see the DHCP discover leave the server with my MAC address, but never see any reply back with my MAC address. I see all kinds of traffic on the wire coming from the cable including APR's and DHCP offers for other users. Its like others make it in but mine don't.
I can spoof my pfSense MAC address on a XP VM in the same ESXi host, in the ESXi host disconnect the WAN interface from my pfSense VM, connect the WAN interface to the XP machine and it instantly acquires a public IP address from the cable company and the connection works fine to the Internet. This at least verifies my cable modem is not locked to some other MAC.
Any ideas? I will continue searching for a solution.
-
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
-
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
I just tried setting promiscuous mode from "reject" to "accept", didn't make any difference.
I don't understand how VMware could be the problem, WAN1 and WAN3 are on the same vinterface on the same vswitch, the interface VLAN is set "VLAN ID: All (4095)", WAN3 works flawless while WAN1 does not, and WAN2 works flawless though it is static IP.
-
When you tested the connection using the WinXP VM how did you handle the VLANs?
A possibility that occurs to me is that untagged traffic is being sent to WAN1 for some reason. This can cause problems with FreeBSDs VLAN handling that may not appear in WinXP.
You should be bale to see this in packet captures.Can you handle the VLAN untagging in VMware instead?
Steve
-
All I know about VMware and its internals would go into a short paragraph so this reply involves a certain amount of speculation.
Your WAN1 uses non-default MAC address. It has been reported in these forums that some FreeBSD LAN drivers do not correctly program the hardware for a change in MAC address. Perhaps pfSense has not issued the correct incantation to change the MAC address on the WAN1 emulated hardware, consequently VMware doesn't know to deliver the DHCP response (to the non-default MAC address) to the WAN1 interface.
The port monitor in the switch shows the cable modem returning DHCP responses. Can you do packet capture in VMware to verify DHCP responses are coming in off the wire? Can you do packet capture in VMware to verify DHCP responses are delivered to pfSense emulated interface? Can you do packet capture in pfSense to verify DHCP responses are seen on the "physical" interface?
Can you do packet capture in pfSense to verify DHCP responses are seen on the WAN1 VLAN interface?Do you really need the non-default MAC address on WAN1? Could you try the default MAC address? (Might need to reboot for change to take effect.)
When you tested the connection using the WinXP VM how did you handle the VLANs?
A possibility that occurs to me is that untagged traffic is being sent to WAN1 for some reason.Could the switch be configured such that it doesn't ALWAYS add VLAN tags to traffic received from the cable modem?
@!real:
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
I just tried setting promiscuous mode from "reject" to "accept", didn't make any difference.
The words suggest to me that the setting allows or rejects an attempt by the client to set an interface to promiscuous mode (accept ALL frames regardless of the destination MAC address). They don't suggest a setting to switch the interface between promiscuous and "normal" mode. I suspect you MIGHT need to set the vswitch promiscuous mode to accept AND set the pfSense physical interface into promiscuous mode (e.g. by pfSense ifconfig shell command or . . .).
-
if you're MAC spoofing in VMware, you must enable promiscuous mode. That sounds more like the issue described most recently where the DHCP responses are on the physical wire, but aren't seen in the VM, which means they're getting dropped within ESX. ESX will drop traffic destined to anything other than VM's assigned virtual MAC address if the vswitch isn't in promiscuous mode.
-
It would be interesting to see what ESXi thinks is the MAC address of that interface.
If you're comfortable with using SSH into your ESXi host and running a script you could try this:
http://www.virtuallyghetto.com/2011/05/how-to-query-for-macs-on-internal.html
I haven't tried it but I'm pretty sure tcpdump is also included ESXi.
-
OK, thanks to stephenw10 I have worked around the problem. Something is still going wrong but at least I have IP's on all my WAN interfaces for the first time!
What I did was left the Cisco 2960 interface (that all WAN's attach) trunked but set the native VLAN to 900. Then set WAN1 to the parent pfSense interface (non VLAN) and it instantly picked up a public IP address from the cable system.
So basically I'm sending WAN1 packets untagged, while WAN2 and WAN3 packets are still tagged. WAN2 and WAN3 seem to still work fine but have not thoroughly tested that yet.
Will do more testing tomorrow. I'm going to try spoofing the MAC on my other DHCP WAN interface and see if it quits working.
-
Interesting. The setup you have now is exactly what I would be trying to avoid!
Sending VLAN tagged and non-tagged traffic to the same pfSense interface can cause problems. However it doesn't always by any means. If it's working for you…
An interesting result though, as though the tagging/untagging process is causing packets not to get through.
I know pretty much nothing about vmware outside the concepts. I was suggesting it may be possible to create a virtual switch that accepts the trunk from the Cisco switch and untags the VLAN traffic to three interfaces internally which then feed pfSense. That way you remove the VLAN handling from pfSense.
Steve
-
I know pretty much nothing about vmware outside the concepts. I was suggesting it may be possible to create a virtual switch that accepts the trunk from the Cisco switch and untags the VLAN traffic to three interfaces internally which then feed pfSense. That way you remove the VLAN handling from pfSense.
Steve
Steve, this can easily be done in VMware. I "thought" it would have been easier to just trunk everything on a single interface but has turned out not the case. From what I have seen so far separate interfaces and no tagging seems the best solution. At work I have over 40 virtual servers (mostly Microsoft :-[) and have never had any problem like this with VMware before.
-
There is an 'issue' with FreeBSD and VLAN handling that can cause a problem when you have tagged and untagged traffic on the same interface. I am not familiar with it and I have never run into it myself but I know it exists. It shouldn't be a problem for you since originally you only had tagged traffic on the interface. In another thread I was trying to diagnose a problem and another user found that his switch was sending untagged packets to his pfSense box accidentally. To eliminate that as a possible cause of trouble I was suggesting you move VLAN handling away from pfSense.
However since you are now definitely sending untagged traffic, and it's working, I'm not sure what to suggest! :-\Steve
Edit: Out of interest I looked further into this and can find nothing so now I'm unsure where this advice came from or what it applies to. Certainly general advice within pfSense is to avoid tagged and untagged traffic on the same interface but whether that is due to FreeBSD or VLANs in general I don't know.
-
OK, problem is something to do with MAC spoofing. I spoofed the MAC on WAN3 and it stopped working. Even after clearing the MAC spoofing box it still would not work until a system reboot which got it working again non-spoofed.
One thing I did notice that was odd, while both WAN1 and WAN2 were spoofed (and not working) ifconfig in pfSense down at the bottom showed the VLAN'ed interfaces with their spoofed MAC's, but oddly WAN2 (which I have never spoofed) showed the same MAC as the spoofed WAN1 MAC. WAN3 showed it's last spoofed MAC even though at this time the spoofing box was cleared on the interface web page. After the reboot everything looked as expected, WAN1 showed it's spoofed MAC, while WAN2 and WAN3 showed the VMware interface MAC.
The problems seems to be spoofing and tagging don't work together. I can spoof non-tagged, and can tag non-spoofed, but can NOT spoof tagged.
Anyone else spoofing and tagging together?
-
you can do any of that, just that VMware complicates things because of its MAC restrictions. When you remove MAC spoofing it doesn't go back to the hardware MAC because there is no way to determine what that MAC is until you reboot. When you spoof on a VLAN parent NIC, it sets that for all the VLANs because of other complications with how that functions and the way many NICs work.
-
Why can't you use the default (hardware) MAC address on WAN1?
-
Why can't you use the default (hardware) MAC address on WAN1?
I could, but that is not what I want to do.
In the past I have always used spoofed the same easy to remember MAC towards the cable modem. This way when I want/need to switch internal devices I can simply shut down one, spoof the same MAC and bring up the other device anywhere on the cable modem VLAN without having to go to and rebooting the cable modem time and time again.
-
My experience was MAC spoofing and VLAN tagging does not work together. To work around my problem I set the spoofed MAC address that I wanted my cable modem to see from my WAN interface, inside the VM setup inside VMware and removed the spoofed MAC address from pfSense. This way pfSense sees and uses my spoofed MAC at boot time as if it was a MAC address on a physical NIC. My cable modem sees and locks to my spoofed MAC and all 3 WAN interfaces works correctly on separate VLAN's on the same physical interface.
This setup has been working fine for a week now. I finally have IP's on all three WAN interfaces with the cable modem locked to the MAC address I need it to use.
Thanks to everyone for all your suggestions!