Vmware pfsense and monowall ISP dhcp problems



  • Hi,

    I have a pretty forward and easy config on pfsense (and monowall) on a vmware esxi 5.5.
    One physical interface for the lan and one interface for the wan, no vlans. I configured as described in doc wiki, but the interface on the wan side never gets an ip adress from ISP.

    So I decided to test with  the pcengines board and voila it works without problems.

    What i tried so far:

    • different vnics (flexibel and E1000)
    • different virtual machine versions (7+8)
    • different virtual hardware platforms (other linux32 and freebsd32)
    • pfsense and monowall
    • different settings in vmware, removing of vswitches and rebuilding etc.
    • vm with installed vmware tools and without
    • added physical mac to virtual machine mac
    • changed cables etc.

    ???

    What I see in logs is just "DHCPDiscover" a few times then giving up. Release / Renew brings nothing.

    If i do a tcpdump on the wan interface i see a lot of arp messages and so on, so there seams to be connection, but pfsense&monowall is unable to get an ip adress from ISP. ISP process first offers an ip of 192.168.100.10 to the interface and then bridges the offical ip something like 109.xx.xx.xx to it.

    As i see in the google search, this seams to be a pretty common problem to pfsense/monowall? Read someting about TTL problems but that should be fixed.

    Anyone has a good idea on that?

    Regards
    proliant


  • Banned

    Your doing it wrong :D

    It works and it works really well. If it doesnt get any IP then the interface is not configured as it should be.


  • LAYER 8 Global Moderator

    "ISP process first offers an ip of 192.168.100.10 to the interface and then bridges the offical ip something like 109.xx.xx.xx to it."

    What isp is this? That seems more like a cable modem offering local when it doesn't have isp connection.

    If you say all you see is dhcp discover and no dhcp offer something is wrong.  Unless your in the same part of the world as the issues with ttl I doubt that is related - and was not actually fixed as of yet in pfsense.  There was a fix to the bug, but I do not believe it is part of the pfsense distro, etc.  I compiled for the user copy that had the higher ttl setting.

    Did you sniff the traffic when it worked - what was the ttls on that traffic?



  • Hi johnpoz,

    thanks for your reply. No, but I will check this.

    My ISP is kabelbw in germany and indeed its an modem, concrete an cisco epc3212.

    I am using the lan interface on that modem. Max speed on that interface is 100 Full.

    BTW a laptop connected to that interface gets an ip without problems, its just the vmware version with problems.

    Regards
    proliant

    @Supermule: And what is wrong? :)


  • Banned

    You have misconfigured interfaces in your VM or on your host setup.

    And use bridge mode on your modem.



  • Hi,

    @Supermule: there is nothing to configure, which could be wrong, just speed and vlans. The modem is not configurable, its ISP locked. BTW more constructive replies means something else then "you have misconfigured…" would be appreciated.

    @johnpoz: TTL cant be the problem, because I am using pfsense on a pcengines board, so there should be the TTL problem as well.

    It must be the virtualization layer between. On which FreeBSD version is pfsense based? What I assume, is that there is something in vmware or in the driver (rtl8168) which is causing the problems. What I could try is how an virtualized windows system on the same infra would react, that would bring or not bring the vmware layer in discussion.

    Regards
    proliant


  • LAYER 8 Global Moderator

    "BTW a laptop connected to that interface gets an ip without problems"

    Did you reset your modem - power cycle if you change the mac of the device connecting to it..  If you connect laptop to your modem and it gets public IP..  Then you connect anything else be it a VM or different laptop, desktop, etc. You more than likely need to power cycle the modem between this change.

    Whenever changing the MAC of device connected to cable modem - good idea to power cycle modem.

    "is that there is something in vmware or in the driver (rtl8168) which is causing"

    Flawed logic – if you see the discover go out, how can there be something wrong in the driver/etc. Or anything to do with that box that would prevent the dhcp offer being sent.  Unless your discover was managled, or it never got to the dhcp server you should see an offer back.



  • johnpoz: yes modem reset, all the time. MAC adress reservation doesnt change the problem, even a copy of the hardware mac to the virtual mac doesnt work.

    logic: it must have something to do with the virtualization. That is the only difference.

    in meantime i have installed a old fritzbox and it works perfect. the problem with my hardware pcengines pfsense was a software crash and then, to repair it, i have to open the pcengines remove board etc. etc.

    Thats why i wanted to change from a hardware pfsense to a vmware software pfsense. But this doesnt work.


  • LAYER 8 Global Moderator

    So I have been running pfsense via vm for long time.. I would do a sniff - do you see the discover go out?  But do not get an offer? Do you have a old hub, or switch that supports span ports?

    Connect your nic from your vm host that vm is connected to to hub/switch with span ports.  Connect other pc/laptop to the span port or one of the span ports.  Troubleshoot the dhcp issue.

    What when good dhcp happens with  your fritzbox, and then with pfsense.  Where are you located?  Could it be related to the short TTL that pfsense sends out vs ttl on the dhcpdiscover that fritzbox sends out and where isp dhcp servers are at.

    Have seen this a couple of times now where the dhcp are lots of hops away and pfsense sets TTL of 16 on dhcp discover.

    https://forum.pfsense.org/index.php?topic=51803.0

    If you see that thread I complied new version for that OP with higher ttl, and that fixed his issue.



  • Hi,

    i tried it again, this time vmware 5.5 and pfsense 2.2 –-> SAME PROBLEM  :'(

    I see DHCP Discovers on the WAN Interface but no offers. If i connect a 10 Euro Netgear router or an old fritzbox they are able to get an Ip Adress.  --> FRUSTRATING :'(

    The outside interface is connected to a cisco modem, the modem first communicates with the outside interface with 192.168.100.1 and then bridges the official ip to the outside interface.
    This means DHCP Discover, DHCP Offer 192.168.100.10 to the pfsense interface, after that offical ip to the pfsense interface.
    pfsense is not able to catch that bridged offical ip adress.
    On the netgear router I see the process->1. ip of 192.168.100.10 2.no ip 3.official ip

    Tried everything:

    not to enable BOGUS and internal interfaces
    switch off firewall
    configure different interface types in vmware (flexibel and e1000)
    and many more.....

    I think no one on the unity media network is able to run pfsense virtualized without an fixed offical ip adress.

    This consumes hours and hours.

    Giving UP Sorry.  :'(

    Some errors in log:

    php-fpm[86165]: /interfaces.php: The command '/sbin/route change -inet default 'XXX.193.32.1'' returned exit code '1', the output was 'route: writing to routing socket: No such process route: writing to routing socket: Network is unreachable change net default: gateway XXX.193.32.1 fib 0: Network is unreachable'
    Feb 12 10:03:45 php-fpm[86165]: /interfaces.php: ROUTING: setting default route to XXX.193.32.1

    Feb 12 10:09:56 dhclient[76130]: DHCPDISCOVER on le1 to 255.255.255.255 port 67 interval 5
    Feb 12 10:09:54 dhclient[76130]: DHCPDISCOVER on le1 to 255.255.255.255 port 67 interval 2
    Feb 12 10:09:53 dhclient[76130]: DHCPDISCOVER on le1 to 255.255.255.255 port 67 interval 1
    Feb 12 10:09:52 dhclient[76130]: DHCPDISCOVER on le1 to 255.255.255.255 port 67 interval 1
    Feb 12 10:09:46 dhclient[76130]: DHCPREQUEST on le1 to 255.255.255.255 port 67
    Feb 12 10:09:42 dhclient[76130]: DHCPREQUEST on le1 to 255.255.255.255 port 67
    Feb 12 10:09:39 dhclient[76130]: DHCPREQUEST on le1 to 255.255.255.255 port 67
    Feb 12 10:09:37 dhclient[76130]: DHCPREQUEST on le1 to 255.255.255.255 port 67
    Feb 12 10:09:37 dhclient: PREINIT

    php-fpm[72358]: /interfaces.php: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf le1 > /tmp/le1_output 2> /tmp/le1_error_output' returned exit code '15', the output was '


  • LAYER 8 Global Moderator

    I have seen it give out the 192.168.100 address - but only when the isp connection is not up yet on the modem.

    So where is the offer, I see the discover - but no offer, what is in the request?  Can you post that up as pcap so can open in wireshark and look at the details.. But without an offer not much you can do.



  • yes its very strange, i didnt see any offers on that interface.

    but, the funny thing is, it worked a few days, then there was a power interruption and after that, pfsense is not able to get an ip adress any more.
    this was happening before as well. my fuse is very weak so sometimes it switches off.

    i have found other articels in forums from people with the same problem and it seams i am not the only one.

    a pfsense in hardware does not have those issues.

    strange..


  • LAYER 8 Global Moderator

    So I have been running pfsense for quite some time on esxi, I have had zero issues with getting ip from isp…  And have moved the mac address to multiple different vms.  I even bring up the mac on other vm router distros to test and no issues.

    I would do a release and renew in pfsense once you have a good isp connection.  You should really never see a 192.168.100 from your modem unless it doesn't have a connection to the isp.



  • It is common to see 192.168.100.1 from a cable modem while the modem hasn't received an IP address from the CMTS (Cable Modem Termination System). I believe your issue might be at a lower level. Here are some assumptions, You have not setup VLANS on ESXi? Check your WAN interface status just because the cable linked up doesn't mean you have all the wires correct if you get the transmits right and not the receives your link light will come on but will not work. If the wire is right then I would try hard code the NIC for the appropriate speed and duplex. PfSense in a ESXi VM is pretty straight forward and works quite well though I prefer to run on dedicated hardware.



  • no i didnt set any vlans. what i see on the netgear device is exactly like that, first 192.168.100.10 then disconnect, then reconnect with official ip (it bridges it i think)
    yes, its really an disconnect there. I hardwired all the connections all ready 1000 FULL, 100 Full etc.
    doesnt work. Changed cable etc. etc.

    strange strange….

    i think the layer 1 theory could be right... but at the moment dont know how to cope with it...



  • That sounds like what happens when the modem itself is issuing you a private IP lease when it's disconnected from upstream. Putting in the IP/subnet the modem assigns in that circumstance on the WAN in the "Reject Leases From" box, which sounds like it should be 192.168.100.0/24.



  • quick update

    I did an install of an virtual smoothwall and all works fine… :o

    So its has something to do with pfsense for sure....


Log in to reply