Can't DHCP from Cable modem
-
Running pfSense 2.0.1 in ESXi 4.1 VMware, I have 3 WAN's on single interface on 3 VLAN's,
WAN1 el0-VLAN900 > DHCP cable modem
WAN2 el0-VLAN100 > Static microwave
WAN3 el0-VLAN200 > DHCP microwaveProblem is pfSense will never get DHCP from the cable modem on WAN1. It DHCP's perfectly from WAN3, and static WAN2 works flawless.
I have powered on and off in every order I can think of. Always power modem off when switching from device to device. All ignore bogon boxes are unchecked. Log shows dhclient is asking for IP but fails with dhclient error "No DHCPOFFERS received." I can do a packet capture on WAN1 and see all the DHCP and ARP requests for everyone else on the cable system. The pfSense charts around 25kbs-35kbs data inbound on the WAN1 interface and appears to be all the broadcast traffic from the cable system.
I can power off modem, plug laptop to modem, power on modem, laptop gets public IP (NOT private IP) and works flawless.
After pfSense tries to get an IP, I can (without powering down the modem) plug laptop into the modem with static IP=192.168.100.5/24 GW=192.168.100.1/24, then access the modem web page 192.168.100.1 and see that it has captured the correct MAC address entered in the psSense WAN1 interface page.
Running SuddenCrap cable, or Suddenlink as they call themselves, cable modem is Cisco 2100 (which this one apparently only works in bridge mode).
Any idea what is going wrong? Could the firewall somehow blocking the DHCP packets on the cable interface? I feel sure it is something simple I'm looking over. ::)
-
@!real:
Could the firewall somehow blocking the DHCP packets on the cable interface?
Have you checked the firewall log? What firewall rules do you have on WAN1?
Could VMware be filtering the response?
-
@!real:
Could the firewall somehow blocking the DHCP packets on the cable interface?
Have you checked the firewall log? What firewall rules do you have on WAN1?
Could VMware be filtering the response?
I see no logs at all for WAN1, log shows WAN3 blocking lots of stuff.
I have no rules under any of the WAN interfaces. WAN3 with no rules DHCP's fine, WAN1 with no rules does not.
I don't think the VMware can be filtering anything. All WAN's are on a single interface on separate VLAN's, same setup for WAN1 and WAN3 and WAN3 works fine.
Doing a packet capture in pfSense and downloading into Wireshark I see my pfSense do 12 DHCP Discover, but never see any DHCP Offers to my pfSense, I do see DHCP offers for others on the network.
Exactly where in the process is the pfSense packet capture taken? Before or after the firewall rules? If a packet is blocked by the firewall will it show up in the packet capture?
-
Not firewall rule related (unless you're putting in some floating rules overriding the default rules). Cable ISPs commonly won't assign a different device an IP, for some just power cycling the modem suffices, for others you have to either spoof the MAC of whatever device had a lease on there previously, or call them. Seriously doubt it's anything other than that based on your description.
-
One other thing came to mind - any MAC your modem sees may be considered a device connected. For instance I have two public IPs via DHCP with my cable provider, and that means they only allow the first two MAC addresses seen. I ran into troubles when I had a managed switch between the cable modem and my firewalls because something the switch was emitting (STP IIRC, it's been a while) was being picked up by the modem as one of my MACs. So the switch was counted as one of my devices even though it wasn't pulling an IP from them. At one point I got someone with a clue on the phone and they upped it to 3 for me so I could actually use two devices, and that got reset at some point and I couldn't get anyone with a clue so ended up just having to put an unmanaged switch in there. If your ESX host is talking on that network (shouldn't unless it has a management IP or similar on that vswitch) or you have a managed switch, you could be hitting the same thing.
-
@!real:
Exactly where in the process is the pfSense packet capture taken? Before or after the firewall rules? If a packet is blocked by the firewall will it show up in the packet capture?
I believe packet capture is before firewall processing on input to pfSense and after firewall processing on output from pfSense.
I presume you have some sort of VLAN capable switch between the WAN1 cable modem and the pfSense VM. Is that correctly configured - for example, is the cable modem port a member of the correct VLAN? Does the switch have a monitor capability (e.g. direct traffic on a VLAN or a port to a 'monitor' port)? If so, you might be able to use your laptop to see traffic on the VLAN of which the cable modem is a member.
Maybe you could get get some help from the ISP tech support: are they seeing your DHCP requests? are they favourably responding to them?
Does the cable modem have some sort of counter you could look at to verify it is getting the DHCP requests from pfSense - e.g. broadcast packets received on the Ethernet interface?
-
@cmb:
One other thing came to mind - any MAC your modem sees may be considered a device connected. For instance I have two public IPs via DHCP with my cable provider, and that means they only allow the first two MAC addresses seen. I ran into troubles when I had a managed switch between the cable modem and my firewalls because something the switch was emitting (STP IIRC, it's been a while) was being picked up by the modem as one of my MACs. So the switch was counted as one of my devices even though it wasn't pulling an IP from them. At one point I got someone with a clue on the phone and they upped it to 3 for me so I could actually use two devices, and that got reset at some point and I couldn't get anyone with a clue so ended up just having to put an unmanaged switch in there. If your ESX host is talking on that network (shouldn't unless it has a management IP or similar on that vswitch) or you have a managed switch, you could be hitting the same thing.
No floating firewall rules. My setup is currently pretty much factory install with the exception of the 2 extra WAN interfaces.
I do have a managed Cisco 2960 that all my WANs go through. I too have had the managed switch blues where it grabbed the only MAC but have solved that a long time ago with other setups. Plus I can view the modem's web page and verify the modem has locked onto my pfSense MAC address and not something else. The modem seems to lock to anything else with just a power cycle, a laptop or WRT54GL router.
Odd thing is I had this same setup working flawlessly several months back on the exact same switches, same server, same modem, same ESXi VMware, until a power failure corrupted everything and pfSense came back up with all new different interface names, forcing a reinstall. I remember having this problem when I built the first setup but can't remember what I did to solve it.
FYI: This config will keep a Cisco switch from taking a MAC from the modem. This will keep the Cisco switch from speaking until spoken too, and has worked great for me in the past.
interface GigabitEthernet0/1
description WAN from Cable Modem
switchport access vlan 900
switchport mode access
no keepalive
no cdp enable
spanning-tree portfast
spanning-tree bpdufilter enable
end -
@!real:
Exactly where in the process is the pfSense packet capture taken? Before or after the firewall rules? If a packet is blocked by the firewall will it show up in the packet capture?
I believe packet capture is before firewall processing on input to pfSense and after firewall processing on output from pfSense.
I presume you have some sort of VLAN capable switch between the WAN1 cable modem and the pfSense VM. Is that correctly configured - for example, is the cable modem port a member of the correct VLAN? Does the switch have a monitor capability (e.g. direct traffic on a VLAN or a port to a 'monitor' port)? If so, you might be able to use your laptop to see traffic on the VLAN of which the cable modem is a member.
Maybe you could get get some help from the ISP tech support: are they seeing your DHCP requests? are they favourably responding to them?
Does the cable modem have some sort of counter you could look at to verify it is getting the DHCP requests from pfSense - e.g. broadcast packets received on the Ethernet interface?
I can span a port on the switch and sniff the traffic. I'm not physically at the switch, its at the other end of a piece of fiber so I have to go to the switch. Not too hard, but not like its here in the basement. That was why I was asking about the pfSense packet capture, trying to figure out if it may be missing something. Probably best to span a port on the switch and watch it to be sure I'm seeing all the packets.
The web interface on this modem is super simple, not much there at all.
I hate the thought of even trying to find any intelligence at their support. Every time I call all I get is "have you tried turning off and back on yet…"
-
@!real:
[Odd thing is I had this same setup working flawlessly several months back on the exact same switches, same server, same modem, same ESXi VMware, until a power failure corrupted everything and pfSense came back up with all new different interface names, forcing a reinstall. I remember having this problem when I built the first setup but can't remember what I did to solve it.
[/quote]
Any chance you have a backup of previous pfSense configuration?Presumably you also lost your VMware config. Is that backed up?
-
@!real:
[Odd thing is I had this same setup working flawlessly several months back on the exact same switches, same server, same modem, same ESXi VMware, until a power failure corrupted everything and pfSense came back up with all new different interface names, forcing a reinstall. I remember having this problem when I built the first setup but can't remember what I did to solve it.
[/quote]
Any chance you have a backup of previous pfSense configuration?Presumably you also lost your VMware config. Is that backed up?
Unfortunately I never backed up the original pfSense install. The ESXi didn't crash, it is the same setup I had before, I just created a new VM. I had first tried to reset my original install to factory defaults but it seemed the OS was hosed as the interface names were all screwy, so I just started over.
-
I would try to verify the DHCP request is at least getting onto the cable plugged into the modem and look for DHCP responses there.
That you created a new VM for pfSense suggests you may not have precisely recreated the VM configuration used by the working pfSense This may not be significant unless the packet capture at the switch (or on the cable to the modem) shows both DHCP request and response.
-
I believe packet capture is before firewall processing on input to pfSense and after firewall processing on output from pfSense.
yes, it's what is on the wire, with the only exception being checksums if you have hardware checksum offloading (in which case the checksums will be 0).
-
OK, spanned a port on the switch and used Wireshark to sniffed the traffic in and out of the WAN interface on the VM host.
I see a DHCP discover leaving my pfSense with the correct MAC address, then immediately see the cable provider reply with a DHCP offer containing my MAC address, but my pfSense never replies with a DHCP request.
Using the pfSense packet capture, I see the DHCP discover leave the server with my MAC address, but never see any reply back with my MAC address. I see all kinds of traffic on the wire coming from the cable including APR's and DHCP offers for other users. Its like others make it in but mine don't.
I can spoof my pfSense MAC address on a XP VM in the same ESXi host, in the ESXi host disconnect the WAN interface from my pfSense VM, connect the WAN interface to the XP machine and it instantly acquires a public IP address from the cable company and the connection works fine to the Internet. This at least verifies my cable modem is not locked to some other MAC.
Any ideas? I will continue searching for a solution.
-
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
-
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
I just tried setting promiscuous mode from "reject" to "accept", didn't make any difference.
I don't understand how VMware could be the problem, WAN1 and WAN3 are on the same vinterface on the same vswitch, the interface VLAN is set "VLAN ID: All (4095)", WAN3 works flawless while WAN1 does not, and WAN2 works flawless though it is static IP.
-
When you tested the connection using the WinXP VM how did you handle the VLANs?
A possibility that occurs to me is that untagged traffic is being sent to WAN1 for some reason. This can cause problems with FreeBSDs VLAN handling that may not appear in WinXP.
You should be bale to see this in packet captures.Can you handle the VLAN untagging in VMware instead?
Steve
-
All I know about VMware and its internals would go into a short paragraph so this reply involves a certain amount of speculation.
Your WAN1 uses non-default MAC address. It has been reported in these forums that some FreeBSD LAN drivers do not correctly program the hardware for a change in MAC address. Perhaps pfSense has not issued the correct incantation to change the MAC address on the WAN1 emulated hardware, consequently VMware doesn't know to deliver the DHCP response (to the non-default MAC address) to the WAN1 interface.
The port monitor in the switch shows the cable modem returning DHCP responses. Can you do packet capture in VMware to verify DHCP responses are coming in off the wire? Can you do packet capture in VMware to verify DHCP responses are delivered to pfSense emulated interface? Can you do packet capture in pfSense to verify DHCP responses are seen on the "physical" interface?
Can you do packet capture in pfSense to verify DHCP responses are seen on the WAN1 VLAN interface?Do you really need the non-default MAC address on WAN1? Could you try the default MAC address? (Might need to reboot for change to take effect.)
When you tested the connection using the WinXP VM how did you handle the VLANs?
A possibility that occurs to me is that untagged traffic is being sent to WAN1 for some reason.Could the switch be configured such that it doesn't ALWAYS add VLAN tags to traffic received from the cable modem?
@!real:
I can't see why it would be needed but have you enabled promiscuous mode on the vswitch?
I just tried setting promiscuous mode from "reject" to "accept", didn't make any difference.
The words suggest to me that the setting allows or rejects an attempt by the client to set an interface to promiscuous mode (accept ALL frames regardless of the destination MAC address). They don't suggest a setting to switch the interface between promiscuous and "normal" mode. I suspect you MIGHT need to set the vswitch promiscuous mode to accept AND set the pfSense physical interface into promiscuous mode (e.g. by pfSense ifconfig shell command or . . .).
-
if you're MAC spoofing in VMware, you must enable promiscuous mode. That sounds more like the issue described most recently where the DHCP responses are on the physical wire, but aren't seen in the VM, which means they're getting dropped within ESX. ESX will drop traffic destined to anything other than VM's assigned virtual MAC address if the vswitch isn't in promiscuous mode.
-
It would be interesting to see what ESXi thinks is the MAC address of that interface.
If you're comfortable with using SSH into your ESXi host and running a script you could try this:
http://www.virtuallyghetto.com/2011/05/how-to-query-for-macs-on-internal.html
I haven't tried it but I'm pretty sure tcpdump is also included ESXi.
-
OK, thanks to stephenw10 I have worked around the problem. Something is still going wrong but at least I have IP's on all my WAN interfaces for the first time!
What I did was left the Cisco 2960 interface (that all WAN's attach) trunked but set the native VLAN to 900. Then set WAN1 to the parent pfSense interface (non VLAN) and it instantly picked up a public IP address from the cable system.
So basically I'm sending WAN1 packets untagged, while WAN2 and WAN3 packets are still tagged. WAN2 and WAN3 seem to still work fine but have not thoroughly tested that yet.
Will do more testing tomorrow. I'm going to try spoofing the MAC on my other DHCP WAN interface and see if it quits working.
